EEP 60: Introduce Support for Experimental Features #

Abstract #

This EEP provides a suggestion on how to allow support for enabling and disabling language features. This will, e.g., allow users to try out, comment on, and suggest changes to new or proposed language features before they are made final. It will also make it possible to avoid using a new language feature, thus making it possible to transition slower, e.g., on a file by file basis, when a new backwards incompatible feature is introduced. Currently this is mainly focused on changes in the language itself, but it can be used for changes in, e.g., the runtime as well.

Rationale #

Other languages support the possibility to experiment with different language features. See the Erlang Experimental Language Features Investigation Report by Kjell Winblad for some examples. This report is included verbatim as an appendix.

Motivation #

We want the possibility to evolve the Erlang/OTP by adding new constructs and remove or change the semantics of existing constructs. Before finalising a new feature as part of the language, it is convenient to allow users to try out a feature in a larger scale without having to run a separate branch of Erlang/OTP. This will enable easier testing of new features as well as facilitating feedback. Before a new feature becomes a permanent part of Erlang/OTP it should be possible to select it for use.

Along the same lines, it should be possible to not use a new feature, especially a feature that changes the semantics of, or removes, an existing construct. The advantage of this is that a transistion of a code base is not forced upon the user when upgrading OTP, but it can be done in a more timely manner.

This will also lead to a uniform way of documenting and introducing experimental features at all levels, i.e., language, runtime, applications and APIs.

We want the possibility to control which features are enabled or disabled during compilation and in the runtime. This control can be made possible by options to the compiler/runtime as well as directives in the module being compiled. Good error messages can then be emitted when required features are not present.

The life cycle of a feature can be seen in the diagram below.

  +--------------+       +----------+
  | Experimental ------> | Approved |
  +---.-------.--+       +-----.----+
      |       |                |
      |       |                |
      |       |                |
      |       |                |
+-----v----+  |          +-----v-----+
| Rejected |  +--------> | Permanent |
+----------+             +-----------+

In addition to the possibility of enabling and disabling a feature, a feature can be enabled by default. There will also be various ways of getting information about features (details below). Some of this is summarised in the following table.

State	Default	Controllable	Available
Experimental	disabled	yes	yes
Approved	enabled	yes	yes
Permanent	enabled	no	yes
Rejected	disabled	no	no

Notes:

• Being controllable means the possibility to enable or disable by means of compiler options and directives in the module being compiled.
• Being available can be seen using the preprocessor macro FEATURE_AVAILABLE and with functions in the erl_features module.

Details #

Modifications for enabling and disabling features or getting information about features need to be added to at least the following modules or general areas:

• erlc - options handling
• erl_scan - handling of keywords
• epp - handling directives and changes to keyword set
• erl_parse (possibly) - new grammar rules for specific features
• The preprocessor
• erl_lint - changes for specific features
• erl_expand_records - changes for specific features
• beam_asm
• Functions in the compile module for handling options
• The -feature(..). directive in a module
• The runtime system - option handling and module loading
• A special module holding the current status of features
• erl_eval - changes for specific features
• Parse transforms

A new module erl_features will be added to provide detailed information about features as well as support functions for feature handling.

In the following, we use some non existing features in the examples. These are, without further explanations at this point, the features maybe_expr (see EEP49), module_alias and ieee754float.

Options to erlc #

The compiler wrapper erlc shall be extended with four new options, two for enabling and disabling features and two for getting information about features. The first two both take a feature name, being an atom, as argument. Several instances are allowed.

1. -enable-feature <feature-name> Turns on the selected feature.
2. -disable-feature <feature-name> Turns off the seleced feature.
3. -list-features Present a list of current features and a short description.
4. describe-feature <feature-name> Present a longer description of the feature.

Additional + style options will be understood by the compiler.

1. +'{enable_feature,<feature-name>}' – see above.
2. +'{disable_feature,<feature-name>}' – see above.
3. +warn_keywords – generate warnings for used atoms that are keywords in some existing feature.
4. +nowarn_keywords – prevent warnings as above.

• The default is to not warn about atoms that are keywords in an existing available feature.
• For both enabling and disabling features, it is possible to use the atom all to enable or disable all available features.

Note: An alternative to the first two + options is a tuple with three elements, i.e., {feature, enable | disable, <feature-name>}. This has the advantage of being more consistent with the format of the -feature directive (see below).

Examples #

• erlc -enable-feature module_alias indicates that the feature module_alias is to be used when compiling the file.
• erlc -enable-feature ieee754float -disable-feature module_alias indicates that the feature ieee754float is be to used when compiling the file, but that the feature module_alias is not. In effect, using an instance of the module_alias should thus generate an error.

Preprocessor Additions #

Add preprocessor macros to enable checking whether a specific feature is available or enabled. We add two predefined macros:

• FEATURE_AVAILABLE(F) – true when feature F is available in the current release. For an unknown feature this will be false.
• FEATURE_ENABLED(F) – true when feature F is enabled at the current location in code. For an unknown feature this will be false.

A use case for having both macros is that one can use FEATURE_AVAILABLE to determine whether a feature is available and, if so, enable it. This will make it easier to write code working in several releases of OTP over a longer time. The macro FEATURE_ENABLED can then be used for code sections with alternative implementations.

Examples #

-if(?FEATURE_AVAILABLE(maybe_expr)).
%% Use the feature when available
-feature(enable, maybe_expr).
-endif.

-if(?FEATURE_ENABLED(maybe_expr)).
%% code that use the feature
-else.
%% alternative code not using the feature
-endif.

%% ..the above also allows simple negative tests
-if(not ?FEATURE_ENABLED(ieee754float)).
..
-endif.

Options to Functions in compile #

Functions in compile that take an options arguments, i.e., file/2, forms/2, noenv_file/2 and noenv_forms/2 should be extended so that the options {enable_feature, atom()} and {disable_feature, atom()} are also recognized.

New -feature(enable|disable, <feature>) directive #

A new -feature(..) directive with two arguments is added. It is only allowed after the -module(..) declaration and in a prefix of the file up to any directives that uses syntax, e.g., a record definition, an -export(..). or a function definition. Preprocessor directives, macro definitions and includes are allowed, but the prefix will end if any of these contain/result in any of the above. The prefix concept is extended to cover included files as well, meaning that the prefix can both be active and ended in an included file.

If the first argument is enable (disable) the feature given by the second argument is enabled (disabled) for the module being compiled.

Several instances of the -feature directive are allowed. An instance of -feature directive in a module will take precedence over options given to the compiler. In effect, enabling and disabling features will have a last write wins semantics.

When compiling a module, a feature will be considered to be used when it has been enabled, even when there are no actual uses of the feature in the module.

Options to the runtime #

Similar to the options to erlc for compiling a module, when starting the runtime, e.g., with erl, we should be able to specify which features we allow. This means that when loading a module it can be rejected due to using, i.e., being compiled with, a feature we do not allow. The reasoning behind this is that one might want to allow features during testing and development, but be more careful about allowing them in production.

The options should be named the same as the options to erlc, i.e., -enable-feature and -disable-feature.

It is not possible to change the set of enabled features after startup.

Informational Module #

A module named erl_features is used to obtain information about the status of known features.

New functions to get information about features:

Get Features available in current release #

features() -> [atom()]

Get Information about a given feature #

feature_info(atom()) -> FeatureInfoMap
when
Description :: string(),
Type :: extension | backwards_incompatible_change,
FeatureInfoMap ::
   #{description := Description,
     short := Description,
     type := Type,
     keywords := [atom()],
     experimental => Release,
     approved => Release,
     permanent => Release,
     rejected => Release,
     status := experimental
             | approved
             | permanent
             | rejected
     }
Release :: non_neg_integer()

%% As above, but give the feature info for a given release
feature_info(atom(), Release) -> Result

Description of keys:

• description - detailed description of feature.
• short - short, one liner blurb, describing feature.
• type - the nature of the feature, i.e., a conservative extension or backwards incompatible change.
• keywords - new keywords introduced by the feature.

• status - the current state of the feature, each state having a corresponding key stating when the feature entered that state.

  Note that all of the keys experimental, approved, permanent and rejected will not be present, but only those up to the current state as seen in the life cycle diagram above.

This can be of use for internal and external tools to:

• Warn about usage of a feature that will be removed.
• Give an error when trying to use a feature that is not present (anymore)
• Tell the compiler which options it needs to use to activate a certain experimental feature
• Automatically remove or warn about instances of the -feature(..) directive when it is no longer neccessary
• Automatically give information about which features have become permanent (or approved) between two releases. This can be used to give the user information about what needs to be changed before an upgrade is made.

The erl_features module will also contain support functions for the actual handling of features, e.g., dynamic keyword handling, but these are implementation dependent and for internal use, so will not be further documented here.

Implementation Notes #

• Since we have the feature options present both in the compiler frontend and the runtime, a compiled module needs to indicate which features have been allowed (or used). This will be done by recording the features used in a new chunk named Meta in the beam file. Upon a load attempt in the runtime, the used features will be checked against the features enabled in the runtime. If the module to be loaded uses features that are not enabled, the load will be disallowed.
• While it might be possible to implement a new language feature only in the frontend, i.e., in essence by a high level macro or parse transform, and thus be rather confident it doesn’t have any affects on later stages of the compiler or runtime (as long as the transformed code is correct), we do not record that level of granularity of feature implementation. We might, after all, change the implementation at a later stage.

Examples #

Support for the new maybe .. else .. end expression (as described in EEP49) will be implemented using the feature mechanism. This will be done using the feature name maybe_expr and it will initially have the status experimental. This will give the community a good opportunity to try it out and give feedback before making it a permanent part of the language.

When compiling a module that uses maybe, the feature maybe_expr needs to be enabled. This can be done in several ways:

1. Use the option to erlc, i.e., erlc -enable-feature maybe_expr
2. Use the possibility for + options, i.e., erlc +'{enable_feature,maybe_expr}'
3. Use a directive in the module being compiled, i.e., -feature(enable, maybe_expr).

To ease the transition of a code base or allow the use in (earlier) releases where the feature is not available, one can use the introduced macros. Enabling of the feature can be done using the compiler options described above. Alternatively, with the code below, one can enable the feature if use_maybe is defined.

 -ifdef(use_maybe).
 -feature(enabled, maybe_expr).
 -endif.

 -if(?FEATURE_ENABLED(maybe_expr)).
 %% Code using the maybe expression
 foo(..) ->
    maybe
    X ?= ..
    end.
 -else.
 %% Alternative (old?) implementation not using maybe
 foo(..) ->
   ..
 -endif.

If the module is compiled with maybe_expr enabled, this will be recorded in the beam file (in the new Meta chunk). To allow loading of the module in the runtime, the feature maybe_expr must be enabled using the enable-feature option.

Backwards Compatibility #

Some possible scenarios in terms of different OTP releases and modules with features:

• A module compiled with a feature enabled and recorded in the Meta chunk can be loaded into a (presumably older) OTP release that does not know about the existence of the chunk. There might be other things preventing the loading, e.g., the use of a new BEAM instruction.
• A module containing -feature directives cannot be compiled by a (presumably older) OTP release that does not know about features. Since the format (with two arguments) is different from that of attributes (one argument) an error will be generated.

Reference Implementation #

An implementation is currently ongoing. The following is currently supported:

• long options to erlc
  • -enable-feature ..
  • -disable-feature ..
• + options to erlc, e.g., +'{enable_feature, maybe_expr}'
• Using +warn_keywords and +nowarn_keywords to erlc is understood. Warnings generated from erl_lint.
• Inline compiler directives
  • -feature(enable, ..).
  • -feature(disable, ..). These are only allowed in a defined prefix of a file.
• Handling of features options in compile:file/2
• Macros FEATURE_AVAILABLE and FEATURE_ENABLED, both of arity 1. The macro FEATURE_ENABLED is changed dynamically when the set of features enabled changes when seeing instances of the -feature directive.
• Dynamic changing of the set of keywords (reserved words), i.e., reacting to the options above for enabling/disabling features.
• Some error handling in detecting and reporting attempted use of currently unknown features.
• Most of the erl_features module, both with functions described in this document and support functions for handling changes in set of keywords (reserved words).
• A new chunk (currently named Meta) is added to the beam file when compiling a module. Thus chunk contains information about the features used (as seen by options set, not if the features was actually present) when the module is compiled. The chunk can be extended to include other meta information in the future.
• Giving options to erl to enable and disable features is possible. Modules that use features that have not been enabled will not load into the system.
• Warnings about atoms that are keywords for existing features can be given (using +warn_keywords to erlc).

Future Work #

• Provide a guide for where to implement different support for a new feature and how one can access the setup of features required.
• Similar to the -std=.. option to gcc specifiying which language standard to compile with one could add a similar option to erlc etc. In short that would be a way to name the collective of all language features being enabled by default in a specific release. Naming the option -lang using -lang=otp24 would mean that we want to compile the input file with all, and only those, features enabled by default in OTP24, even if erlc is from the OTP25 release. Any features added in OTP25 would thus not be allowed. It would be allowed, though, if one added the enable-feature option, e.g., erlc -lang=otp24 -enable-feature module_alias

References #

Appendix #

This is the original report by Kjell Winblad report. Some of this should be copied elsewhere to the document as it provides a good background with regards to other languages and set the foundation.

Erlang Experimental Language Features Investigation Report #

Erlang currently does not support selectively using experimental language features that are not officially part of the Erlang language. Having support for doing so can help users try out and experiment with a potential extension to the language without adding this extension to the main language. This report looks into how support for selectively including experimental language features looks in other languages and how such support might look in Erlang.

Support for Selectively Enabling Experimental Language Features in Other Languages #

Python #

Pyhton has support for making language extensions or changes optional before they become mandatory. The Python module __future__ defines several feature names like this:

FeatureName = _Feature(OptionalRelease, MandatoryRelease,
                       CompilerFlag)

• OptionalRelease is the release in which one could first optionally enable the feature
• MandatoryRelease is the release when the feature became/”is planned to be” mandatory
• CompilerFlag is the compiler flags that need to be passed to the compile module function to enable the feature

Python has a special statement that needs to be placed near the top of a python module to enable a language feature in a specific module. Statements to enable features that have already become mandatory have no effect.

In Python, feature names are never removed from the __future__ module, which means that the __future__ module contains a history of language changes.

Some of the benefits of Pythons future import statement and future module are:

• Users can start migrating code module by module before using a release in which a potential backward incompatible change is made mandatory.
• Users can start to experiment with language extensions before they are enabled by default
• A problem with a language extension can be found and fixed (or the language extension can be removed) before it is made mandatory
• A programmatically accessible history of language changes is accessible through the __future__ module. Tools can use this history, for example, to remove from __future__ import x-statements that are no longer necessary.

Ruby #

Ruby does not have special support for experimental features (experimental features are just documented as experimental in the documentation). See this issue that proposes using command line flags to enable experimental features for more information.

Rust #

Rust has a special syntax for activating experimental language features. Here is one example:

#![feature(box_syntax)]

fn main() {
    let five = box 5;
}

Such features are called unstable in Rust’s terminology. They might change or disappear at any time.

One activates the feature for the current compilation unit (crate).

Haskell #

Haskell makes it possible to activate certain language features with a pragma in the file header:

{-# LANGUAGE TemplateHaskell #-}

Java #

Java lets users test features that are planned for a later release. This needs to be enabled by passing compiling flags to the compiler when compiling the Java file:

javac --enable-preview --release 12 # other flags

The line above can enable language features planned for Java version 12 in earlier Java versions before Java version 12 is released. To limit the use of preview features, one also has to pass -enable-preview when running a Java program compiled using the -enable-preview flag. A warning message is always printed when a preview feature is used.

When one enables preview features for a specific release in java, one gets all preview features from that release. It is not possible to select a single feature.

Features are not released as a preview feature unless they are considered good enough to be included without modifications. Therefore, changes to preview features are relatively rare but can happen.

See here for more details.

Suggestion for Erlang #

The following methods may be used to activate an experimental feature:

• -compile(). directive inside a file,
• an option passed to one of the compile functions in the compile module or,
• a compilation flag passed to erlc.

The option/flag to enable an experimental feature can have a prefix and the experimental future’s name:

Examples:

-compile([{enable_experimental, pinning_operator}]).

compile:file(File, [{enable_experimental,pinning_operator}])

erlc -enable_experimental_pinning_operator

In the above examples, enable_experimental is the prefix and pinning_operator is the experimental feature’s name.

All experimental features currently existing and that have existed in the past can be “documented” in a special module (similar to Pyhton). Let us assume that this module is called experimental_features. This module can be public to allow external tools to use the module or internal only if we want to be able to make changes to its API.

The experimental_features module has functions that can be used to obtain information about experimental features:

list_experimental_features() -> [atom()].

This function returns a list of all experimental features that are currently existing and that have existed.

get_experimental_feature_info(FeatureName) -> Result when
FeatureName :: atom(),
description :: string(),
Type :: extension | backwords_incompatible_change,
Result :: missing | FeatureInfoMap,
FeatureInfoMap :: #{optional_release := ReleaseNr,
                status := experimental | %% May be removed
                                         %% or changed
                          {remove_planned, ReleaseNr,
                           AdditionalInfo :: string()} |
                          {inclusion_planned, ReleaseNr} |
                          {removed, ReleaseNr,
                           AdditionalInfo :: string()},
                          {included, ReleaseNr}
                 %% list of compiler options that needs
                 %% to be given to activate this feature
                 %% (this can be useful, for example, when
                 %% one experimental feature depend on another)
                compiler_options := list()
               },
ReleaseNr : {Major :: integer(),
             Minor :: integer(),
             Patch :: integer(),
             Label :: string()}.

This function returns information associated with a given feature name.

External and internal tools may use the information that one can get from the experimental_features module to:

• Warn about the usage of something that will be removed
• Give error if something that has got removed is used
• Tell the compiler which options it should use to activate a certain experimental feature
• Automatically remove {enable_experimental,x} tuples from -compile() directives when they are no longer necessary.
• Automatically give information about which features become mandatory between two releases (which can give the user information about what needs to be changed before an upgrade is made)

Preventing that Experimental Features are used too Much in Production Code #

Similar to Java we can emit information that an experimental feature is used in the compiled module. The VM can use this information to print a warning message when running a module that contains an experimental feature. We can also force usage of a special flag when running code that are compiled with an experimental feature.

Copyright #

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.