The majority of those features are written as elisp libraries. They're only loaded if you use them. Unless you're concerned with disk space (why? Disk space is cheap), they aren't the source of your bloat.
The features that are built-in can be changed via the configure script, assuming you don't mind compiling Emacs.
If Emacs feels sluggish, then you're probably loading unnecessary things in your init.el. Or you're on Windows and your virus scanner is fighting with it (that happens to me on my work laptop with Trend Micro).
Edit: org-mode is fairly large and can take a bit to load the first time. Could that be what you're experiencing?