It is possible to extend the Component API for custom front features.
What is important here is to keep in mind you should do it only if it targets not portable components (only used by the Studio or Beam).
In term of organization it is recommended to create a custom xxxx-component-api module with the new set of annotations.
Extending the UI$
To extend the UI just add an annotation which can be put on @Option fields which is decorated with @Ui.
All its members will be put in the metadata of the parameter. Example:
@Ui
@Target(TYPE)
@Retention(RUNTIME)
public @interface MyLayout {
}