Generating Module Libraries
This section provides further details on module generation. We begin by discussing template substitution, the underlying process by which the generated .c and .h files are created. The section also documents MATLAB commands that are related to module generation. Many of the commands shown were used in previous examples and are fully documented here.
The main phases in creating an audio module library are:
For each audio module in the library
Add code markers
Generate the module's .c and .h files using template substitution
Generate the combined schema file describing all of the modules in the library
Translate the schema information into a C data structure that can be compiled and included with the DLL.
Compile the generated files and create an audio module dynamic link library (DLL).
Steps 1 and 3 are automated using the awe_generate_library.m command. Many other MATLAB functions are internally called. Only a few other functions are documented here to help you better understand the code generation process. Step 4 is done separately in VisualStudio and described in Generating Documentation.
awe_generate_library.m
awe_generate_library(MM, DIR, LIBNAME, USESDLLS, GENDOC)
This is the primary function for generating an audio module library. It generates sources files for all of the modules in the library. Arguments:
MM — cell array of audio module structures.
DIR — base directory in which the generated files should be placed. Files are written here and in several other subdirectories.
LIBNAME — string specifying the name of the library. For example, 'Advanced'.
USESDLLS — structure specifying other module libraries that the newly built library depends upon. A library has a dependencies when it internally utilizes a module contained in another library. By default,
USESDLLS=[]and the library is independent of others.GENDOC — integer indicating whether module documentation should be generated. By default,
GENDOC=0and no documentation is generated. IfGENDOC=1, then a single document is created for the entire module library. IfGENDOC=2, then a separate document is created for each audio module.
Creating the Cell Array of Modules
The argument MM, is a cell array populated with modules to be placed in the final library. For example, it might contain:
MM=cell(0,0);
MM{end+1}=downsampler_example_module('temp');
MM{end+1}=fader_example_fract32_module('temp');
MM{end+1}=fader_example_module('temp');
MM{end+1}=lah_limiter_example_module('temp');
MM{end+1}=peak_hold_example_fract32_module('temp');
MM{end+1}=peak_hold_example_module('temp');
MM{end+1}=scaler_example_module('temp');
MM{end+1}=scaler_smoothed_example_module('temp');Specifying Dependencies
USESDLLS is a cell array of structures that specifies dependencies. Each structure contains the fields:
.MM – cell array of audio modules in the library.
.str – library name.
To achieve automation, each audio module library supplied with Audio Weaver has a master function which generates the library. For example, the Advanced module library uses the script make_advancedmodulepack.m. The script is designed so that if you request output arguments, as in
[MM, NAME]=make_advancedmodulepack;
the function will not build the library but will instead return a cell array of modules in MM and the name of the module library in NAME. You should follow this convention when creating your own custom module libraries. This allows you to specify dependencies in a straightforward manner. For example, the make_examples.m script is dependent upon 3 other libraries:
[DependMM, DependName]=make_standardmodulepack(0);
USESLIB{1}.MM=DependMM;
USESLIB{1}.str=DependName;
[DependMM, DependName]=make_deprecatedmodulepack(0);
USESLIB{2}.MM=DependMM;
USESLIB{2}.str=DependName;
[DependMM, DependName]=make_advancedmodulepack(0);
USESLIB{3}.MM=DependMM;
USESLIB{3}.str=DependName;If you try to build a library and do not properly account for all dependencies, then awe_make_library.m will report an error. For example, suppose that you try and build the examples library but eliminate the dependency on the Standard/Deprecated libraries, you'll see:
Module library has unsatisfiable or circular dependencies.
Modules with dependencies that could not be resolved:
FaderExample
Missing ScalerSmoothed
LAHLimiter
Missing AGCLimiterCore
Missing AGCMultiplier
Missing MaxAbsThe error message indicates that the FaderExample module has a dependency on ScalerSmoothed and that ScalerSmoothed is not in the dependency list. Similarly, LAHLimiter has 3 unsatisfied dependencies.
Specifying the Output Directory
The argument DIR to awe_generate_library.m specifies the directory in which to generate the modules. There are different ways to specify the directory including hard coding it.
Often it is useful to specify the directory relative to the location of the master library make script. The following MATLAB code, contained in make_examples.m, shows how to determine the location of make_examples.m and then pull off the lowest level directory. make_examples.m is located in:
📁 <AWE>\AWEModules\Source\Examples\matlab\
The output directory is set to
📁 <AWE>\AWEModules\Source\Examples\
by the code below.
MFILE=mfilename('fullpath');
[pathstr, name]=fileparts(MFILE);
% Remove the last directory level
ind=find(pathstr == filesep);
ind=max(ind);Generated Files
Let DIR be the base directory for the audio module library and let 'Examples' be the name of the library. awe_generate_library.m creates the following files and folders:
<DIR>\Examples.h — Header file containing extern definitions for each module class object. It also has macros defining the entire list of modules and all dependencies.
<DIR>\Examples.sch — Overall schema file for the library.
<DIR>\ExamplesSchema.cpp — Compiled schema file.
📁 <DIR>\Doc — Location of generated documentation
📁 <DIR>\Include — Location of generated module include files
📁 <DIR>\matlab — Location of the module m-files. This also contains the master library make script.
📁 <DIR>\matlab\code — Suggested location for the inner code pieces.
📁 <DIR>\matlab\process — Suggested location of the MATLAB versions of the processing functions.
📁 <DIR>\matlab\test — Suggested location of MATLAB test scripts which validate the operation of the modules.
📁 <DIR>\Source – Location of the generated module source files.
📁 <DIR>\xml – Module XML file for AWE Designer, explained in get_variable.m and set_variable.m .
SchemaBuilder.exe
You'll notice this executable within the Audio Weaver Bin directory. This executable compiles the schema information. That is, it takes a .sch file and compiles into a .cpp file. The .cpp file is then included in the project for building the module DLL. SchemaBuilder.exe is automatically called by awe_generate_library.m and there is no need to call it separately.
AWE_INFO.buildControl
The global variable AWE_INFO contain the structure .buildControl which controls the library generation process. AWE_INFO.buildControl contains many fields which are used internally. We point out the user settable values.
AWE_INFO.buildControl.combineSourceFiles — this Boolean specifies whether each audio module should be placed into a separate source file. By default,
combineSourceFiles=0and each module is written to a separate file.AWE_INFO.buildControl.indentSourceFiles — this Boolean specifies whether the source code should be formatted after generation using the executable indent.exe. Formatting the code regularizes identing, line breaks, etc. By default,
indentSourceFiles=0. Formatting the source code significantly slows down the module generation process.
Setting the Audio Module Search Path
When you are developing a custom audio module, you must add the base module directory to the Audio Weaver module search path.
Caution: It is not enough to merely modify your MATLAB path.
The path can be updated in Designer using the ‘File->Set Module Path’ menu option. To add a directory to the Audio Weaver module search path programmatically, use the command
add_module_path(DIR)
The command adds the directory
📁 <DIR>\matlab
to the MATLAB search path; this directory has to exist. It then optionally adds
📁 <DIR>\matlab\test
📁 <DIR>\matlab\process
if these directories exist. In addition, the function modifies the global variable
AWE_INFO.buildControl.modulePath
to include the new directory. This global variable is used when searching for class IDs.
The newly added module directory is placed at the end of the module search path. You can specify that the directory be added to either the beginning or end of the module path using the optional second argument
add_module_path(PATH, '-begin');
add_module_path(PATH, '-end');Specifying Class IDs
Every audio module in the library must have a unique class ID. This 32-bit integer is specified in the file
<DIR>\classids.csv
This comma separated text file contains entries of the form
ClassName,ID
The file can also contain a single entry at the start of the form:
IDOFFSET=32768
The IDOFFSET is added to each ID listed in the file. Using IDOFFSET allows you to quickly make wholesale changes to all of the modules in the library.
DSP Concepts reserves class IDs in the range 0 to 32767 for module libraries shipped with Audio Weaver. You are free to use any IDs in the range 32768 to 63487 for your custom modules.
Audio Weaver provides a function for looking up classIDs across a range of audio module libraries.
ID=classid_lookup(CLASSNAME)
You pass the function a class name string and the classID is returned. The function works by using the audio module directory search path contained in
AWE_INFO.buildControl.modulePath
For each directory, the file classids.csv is opened and examined. The function classid_lookup.m uses internal caching to speed up the search process.
Reordering of Render Variables
As part of the build process, Audio Weaver reorders the variables in a module or subsystem to match the manner in which modules are instantiated by the Server. The reordering is performed by the function
M=awe_reorder_variables(M)
where M is an @awe_module or @awe_subsystem object. Variables are ordered according to the rules
All non-hidden scalar variables
All hidden scalar variables
All array variables (pointers)
Pointers to internal modules (subsystems only)
The ordering enforces that all scalar variables, which are initialized by the base module constructor function, are located at the start of the instance structure. Refer to Module Function Details for additional information.
Overwriting Existing Source Files
Audio Weaver is frequently used in conjunction with a source code control system. The awe_generate_library.m function overwrites generated files only if there is an actual change to the file. If nothing has changed, the file is untouched.
Each generated file is first written to a temporary file. Then the temporary file is compared with the existing version. If the files are identical, then the existing file is untouched and the temporary file is deleted. If the temporary file differs from the existing file, then the existing file is overwritten.
Specifying Wiring Constraints
The .wireAllocation field of an audio module allows you to specify wiring constraints for the routing algorithm. There are two possible values
'distinct' — None of the input wire buffers will be used as output wire buffers. That is, the outputs will be "distinct" from the inputs.
'across' — The routing algorithm will try and reuse wire buffers across the module, whenever possible. That is, the wire buffer assigned to input pin N will be reused for output pin N. The wires are the same "across" the module.
awe_module.m sets .wireAllocation to 'distinct' by default. This is the safest approach but consumes more memory. 'across' conserves memory and is often easier to debug. When 'distinct' is specified, the output wires will always be different compared to the inputs. When 'across' is specified, the routing algorithm will attempt to reuse the buffers, but this may not always be the case.
The choice between 'distinct' and 'across' depends upon the internal details of your processing algorithm. Buffer pointer usage within the algorithm has to be studied to determine if 'across' wiring is possible. Consider the scaler module processing function shown below. This module has N inputs and N outputs with the Nth input being scaled and written to the Nth output.
void awe_modScalerProcess(void *pInstance)
{
awe_modScalerInstance *S = (awe_modScalerInstance *)pInstance;
WireInstance **pWires = ClassModule_GetWires(S);
int numPins = ClassModule_GetNInWires(S);
int pin;
for(pin=0;pin<numPins;pin++)
{
awe_vecScale((float *)(pWires[pin]->buffer), 1,
(float *)(pWires[numPins+pin]->buffer), 1,
S->gain, ClassWire_GetNumSamples(pWires[pin]));
}
}It is safe to use 'across' wiring in this case. Now consider the Mixer module. This module has 1 input pin with M channels and 1 output pin with N channels. Each output channel is formed as a weighted sum of input channels with each output channel written in order. In this case, you can no longer use 'across' wiring because each output channel depends upon each input channel. If 'across' wiring were used, then writing the first output channel would overwrite the first input channel.
Note that 'across' is a suggestion to the routing algorithm. In certain cases, 'across' wiring cannot be used and distinct buffers will be provided. You should always write your audio module processing function anticipating that the buffer pointers may be distinct. For example, the inner loop of the awe_modScalerExampleProcess function should not be written as:
for(pin=0;pin<numPins;pin++)
{
awe_vecScale((float *)(pWires[pin]->buffer), 1,
(float *)(pWires[pin]->buffer), 1,
S->gain, ClassWire_GetNumSamples(pWires[pin]));
}The routing algorithm may decide to assign a distinct output buffer; always pass in the given output buffer pointer.
The most common situation when 'across' wiring cannot be applied is when a module sits between the input and output of a system. For routing reasons, the wires attached to the input and output of a subsystem have to be distinct. Placing a module with .wireAllocation='across' between them will still lead to distinct buffer allocation.
AudioWeaverModule Tag
The function awe_help.m displays a list of all Audio Weaver modules on the current module search path. In order for a module m-file to be picked up, you need to add the tag AudioWeaverModule somewhere within the m-file. This tag allows you to differentiate between actual module m-files and other MATLAB files contained in the same directories. The tag can be placed anywhere within the file, usually in a comment as shown below:
% AudioWeaverModule [This tag makes it appear under awe_help]
Code Markers and Template Substitution
Template substitution is an automated method of replacing strings within a file to generate an output file. The process starts with a template file and Audio Weaver provides separate templates for the generated .c and .h files:
<AWE>\matlab\module_generation\templates\awe_module_template.h
<AWE>\matlab\module_generation\templates\awe_module_template.c
A template file contains normal text interspersed with entries of the form
$IDENTIFIER$
where IDENTIFIER is a string. The template substitution function is given a list of identifiers together with their replacements. The substitutions are performed and a new output file is created. If an identifier within a file is not defined, then it is deleted and does not appear in the output.
Adding Code Markers
A "code marker" is an identifier together with a replacement string. Each module and subsystem has its own list of code markers. The MATLAB function
awe_addcodemarker(M, IDENTIFER, SUBSTITUTION)
creates a new code marker. The function arguments are:
M — @awe_module or @awe_subsystem object
IDENTIFIER — string indicating a specific identifier in a template file.
SUBSTITUTION — string containing the replacement value for the identifier.
SUBSTITUTION can be either a single string or a cell array of strings for multi-line replacements. If you call awe_addcodemarker.m and an IDENTIFIER of the specified name already exists, then the new SUBSTITUTION string is appended as another line. For example, the following entry located near the top of awe_module_template.c file is used to specify include files:
$srcFileInclude$
If you call
awe_addcodemarker(M, 'srcFileInclude', '#include "file1.h"');
awe_addcodemarker(M, 'srcFileInclude', '#include "file2.h"');Then the generated file will include both lines:
#include "file1.h"
#include "file2.h"
An optional 4th argument selects between appending and overwriting duplicate code markers.
awe_addcodemarker(M, IDENTIFER, SUBSTITUTION, APPEND)
By default, APPEND=1 and duplicate code markers are appended. If you set APPEND=0, then an existing code marker of the same name is overwritten. For example,
awe_addcodemarker(M, 'srcFileInclude', '#include "file1.h"');
awe_addcodemarker(M, 'srcFileInclude', '#include "file2.h"', 0);Since $srcFileInclude$ is already defined, the second call to awe_addcodemarker.m will overwrite the code marker and the generated file will only contain
#include "file2.h"
The call awe_addcodemarker.m adds the code marker information into the field .codeMarker of the module. You could, for example, examine the code markers defined for the scaler_module.m as follows:
>> M=scaler_module('fred');
>> M.codeMarker{1}
ans =
name: 'processFunction'
text: 'Insert:code\InnerScaler_Process.c'
>> M.codeMarker{2}
ans =
name: 'discussion'
text: {1x9 cell}Inserting Files
In many cases, the SUBSTITUTION string contains many lines and it is unwieldy to represent in MATLAB. Instead, it is easier to copy the SUBSTITUTION text from another file. If the SUBSTITUTION string has the form "Insert:filename.txt" then the text will be copied from filename.txt. filename.txt is referenced relative to the location of the module's m-file. You can specify subdirectories as well. For example:
awe_addcodemarker(M, 'processFunction', ...
'Insert:InnerPeakHold_Process.c');inserts the file InnerPeakHold_Process.c located in the subdirectory "Inner".
awe_lookupcodemarker.m
The MATLAB function
STR=awe_lookupcodemarker(M, IDENTIFIER)
returns the contents of a code marker. If IDENTIFIER is not defined, then the function returns an empty string. The returned value STR is either a string, for simple replacements, or a cell array in the case of multi-line replacements.
awe_deletecodemarker.m
M=awe_deletecodemarker(M, IDENTIFIER)
Deletes the code marker named IDENTIFIER.
M=awe_deletecodemarker(M)
Deletes all code markers associated with a module or subsystem.
Template Substitution Preprocessor
The template files also contain their own preprocessor directives. These directives are similar to those used by the C preprocessor but are handled by the template substitution function. The directives are identified by ## and only a few variations are supported.
##if 1
… This code will appear in the output file …
##endif
##if 0
… This code will NOT appear in the output file …
##endifWhen the argument to ##if is 1, then the text between the ##if and ##endif will appear in the generated file. Otherwise, the text will not appear. You can also use an identifier with an ##if statement as in
##if $myVariable$
Optional text
##endifIf you set
awe_addcodemarker(M, 'myVariable', '1')
then the optional text will appear in the generated file. Note that $myVariable$ is being set to the string '1' not to the numeric value 1. The template preprocessor directive also supports an else clause
##if $combineSourceFiles$
#include "$combinedIncludeName$"
##else
#include "$baseHFileName$"
##endifFrequently Defined Code Markers
This section lists out code markers that are frequently defined when developing custom audio modules.
$bypassFunction$
C code which specifies the inner portion of the bypass function. When this is defined, the bypass function is written into generated C file.
$bypassFunctionName$
Allows you to specify a bypass function by name. This is typically used in conjunction with one of the predefined bypass functions listed in Bypass Function. For example,
awe_addcodemarker(M, 'bypassFunctionName', 'IOAcrossModule_Bypass');
The following logic defines the bypass function.
If
$bypassFunction$is defined, then the code is placed into the generated .c file and$bypassFunctionName$is set toawe_modClassNameBypass,If
$bypassFunctionName$is defined, then this function is used for bypass behavior. No additional code is placed into the generated .c file.Otherwise, the default bypass function
IOMatchUpModule_Bypass()is used. No additional code is placed into the generated .c file.
$constructorFunction$
Specifies the inner portion of a custom constructor function. The function must follow the calling convention outlined in Constructor Function.
Simple audio modules without indirect arrays: No need to define the constructor function. The
BaseClassModule_Constructor()usually does everything needed.Audio modules with direct arrays: Set the
.arrayHeapand.arraySizeConstructorfields of the array variables in MATLAB. Audio Weaver will then auto generate a suitable constructor function.Complex modules requiring further initialization: Write a custom
$constructorFunction$or use the$postConstructorFunction$shown below.Modules created out of subsystems: Audio Weaver automatically generates code for these systems. If you need further initialization, use the
$postConstructorFunction$shown below.
$postConstructorFunction$
This code marker follows immediately after the $constructorFunction$ code marker in awe_module_template.c. This marker allows you to add your own code in cases where Audio Weaver generates the memory allocation and instantiation code but further initialization is needed. It is particularly useful for subsystems.
$discussion$
Defines the discussion section of the help documentation. This has been grandfathered for backwards compatibility. When writing new modules, set the M.docInfo.discussion field of the audio module instead.
$getFunction$
Specifies the inner portion of a module's Get function. Follows the calling convention outlined in Get Function .
$setFunction$
Specifies the inner portion of a module's Set function. Follows the calling convention outlined in Set Function.
$hFileDefine$
Located near the top of the template file awe_module_template.h. Allows you to add additional preprocessor directives (or anything else) at the start of the header file. Example,
awe_addcodemarker(M, 'hFileDefine', '#define TRUE 1');
$hFileInclude$
Located near the top of the template file awe_module_template.h. Allows you to include additional header files.
awe_addcodemarker(M, 'hFileInclude', '#define <stdlib.h>');
$processFunction$
Specifies the inner code for the module's processing function. This must always be defined for modules. For subsystems, the auto-generated should be used. Refer to Processing Function for a discussion of how to define this function.
$preProcessFunction$
This code marker is located immediately before the $processFunction$ marker. It is typically used with subsystems when you want to insert your code custom code immediately prior to the auto-generated $processFunction$.
$postProcessFunction$
Similar to $preProcessFunction$ but the code here is inserted immediately after the $processFunction$. It is typically used with subsystems when you want to insert your custom code immediately after the auto-generated $processFunction$.
$srcFileDefine$
Located near the top of the template file awe_module_template.c. Allows you to add additional preprocessor directives (or anything else) at the start of the source file.
$srcFileInclude$
Located near the top of the template file awe_module_template.c. Allows you to specify additional included files.
$hFileTemplate$ and $srcFileTemplate$
Audio Weaver uses the default template files specified at the start of Code Markers and Template Substitution. You can override the templates used by a particular module by setting these code markers. There are separate markers for the header file, $hFileTemplate$, and the source file, $srcFileTemplate$. It is handy to override these values if you want to change copyright information in the generated files.
Fine Tuning Code Generation
This section contains further instructions for fine tuning code generation. Many of the items apply to modules generated using compiled subsystems.
Automatically Generated Array Initializers
In many cases, an audio module requires a custom constructor function solely for the purpose of allocating indirect arrays. If the size of the allocated memory can be defined using existing module variables, then Audio Weaver can automatically generate a constructor function.
The generated array constructor code is of the form:
if ((S->ARRAYVAR = (TYPE *) awe_fwMalloc(SIZE, HEAP, retVal)) == 0)
{
// Error code is in *retVal
return 0;
}Where:
ARRAYVAR — is the name of the variable
TYPE — is the type of the module (either float, fract32, or int)
SIZE — is a user specified string
HEAP — is a user specified string.
ARRAYVAR and TYPE are already known to MATLAB. SIZE and HEAP must be separately specified as C code to embed into the generated code. The string SIZE is specified in MATLAB by setting .arraySizeConstructor field of the array variable. Similarly, HEAP is specified by setting the .arrayHeap field of the array variable. For an example, see Peak Hold Module.
Specifying Module Constructor Arguments in Subsystems
This advanced technique is frequently used when compiling subsystems to generate new module classes. Audio Weaver automatically generates a Constructor function for the compiled subsystem, and this constructor in turn calls the constructor functions of the internal modules. The difficulty arises because the arguments to the internal constructor functions are hard coded in the generated file based on the values at code generation time. This is not always the desired behavior.
To get around this problem, Audio Weaver allows you to specify a piece of C code which overrides the default numeric initializer for an internal module variable. The C code is written to the .constructorCode field of the variable. For an example, refer to the Look Ahead Limiter Module.
Avoiding Wire Allocation Problems with Subsystems
This issue also applies to the case of compiling subsystems to form new module classes. If your subsystem has .flattenOnBuild=1 (which means that it is not compiled), then you can ignore this section. This problem arises because Audio Weaver modules can operate on an arbitrary number of channels. In certain circumstances, the routing algorithm incorrectly allocates wire buffers during code generation such that audio processing crashes at run-time. In these cases, you need to be careful how you specify default channel counts and block sizes within modules. Look Ahead Limiter Module demonstrates the problem and presents a solution.
unique_classes.m
C=unique_classes(SYS)
where
SYS — @awe_subsystem object.
Returns a list of all the unique module and subsystem classes found within subsystem SYS. The function recursively searches through the system and identifies individual modules and subsystems. The return result C is a cell array of strings. If called without any output arguments, the function displays the class list to the MATLAB output window. The class name of the input SYS is not included in the C. This function is useful for dependency checking.
[C, M]=unique_classes(SYS)
An optional second output receives a cell array of the actual modules/subsystems listed in C.
SYS can also be a cell array of subsystems. In this case, the function returns the overall set of unique classes needed to create all of the systems in SYS.
[C, M]=unique_classes(SYS CLIST)
An optional Boolean argument CLIST specifies that the list of unique module classes should be displayed in a manner appropriate for inclusion in the TargetInfo.h file.
For example, to determine all of the unique module classes needed for the agc_example, type:
>> SYS=agc_example;
>> unique_classes(SYS)
Unique classes within the subsystem of class: test
AGC
AGCCore
AGCMultiplier
Meter
ScalerDBIf you want to build a target with only the modules required for the agc_example, you would set CLIST=1. The following C code is then printed to the output window:
extern const ModClassModule awe_modAGCClass;
extern const ModClassModule awe_modAGCCoreClass;
extern const ModClassModule awe_modAGCMultiplierClass;
extern const ModClassModule awe_modMeterClass;
extern const ModClassModule awe_modScalerDBClass;
#define LISTOFCLASSOBJECTS \
&awe_modAGCClass, \
&awe_modAGCCoreClass, \
&awe_modAGCMultiplierClass, \
&awe_modMeterClass, \
&awe_modScalerDBClassThis text can be pasted directly into TargetInfo.h.
awe_generate_module.m
M=awe_generate_module(M, DIRECTORY, WRITEFILES)
Internal function used by awe_generate_library.m. In some cases, such as when generating documentation, you may want to call this function directly. Arguments:
M — @awe_module object.
DIRECTORY — directory in which the generated files should be written to. By default, this is set to MATLAB's current working directory (returned by the
pwdcommand).WRITEFILES — an optional Boolean argument which specifies whether the generated files should actually be written. If you set
WRITEFILES=0, then the module M will be updated with code markers, but no output files will be written. UseWRITEFILES=0when generating documentation as described in Generating Documentation.