Back in February, I wrote a blog post about how we can use the XBRL Object in Legato to do some extra validation on XBRL. In that blog we added a check for the number of custom elements. That was a relatively basic application of XBRL in Legato though, not very complicated. What if we want to do something a lot more in depth? For example, check to ensure we’re not defining any cyclic presentation relationships. By that, I mean that you have a dimension - member pair in a context, but in the presentation properties for a presentation where that context appears the domain for that dimension is set to the same member. That’s a cyclic relationship; saying something is it’s own parent element. This isn’t Futurama, you can’t be your own grandfather in XBRL.
Friday, May 04. 2018
LDC #83: Validating With The XBRL Object Part 2
To do this, we’re going to need to do a lot more manipulation of the XBRL Object than we did in the last blog. To make things a little more clear, I think it’s a good idea to discuss which XBRL functions we’re going to need before we go over the script. The XBRL functions used in the script are:
XBRLGetObject - We used this one last time too, it just grabs a handle to the XBRL object.
XBRLGetPresentations - Gets an array containing a list of presentation names.
XBRLGetContexts - Gets an array containing a list of context identifiers.
XBRLGetPresentationType - Gets the type of a presentation. You need to pass it the XBRL object and the index of the presentation. See the documentation for return values, but anything over 500 is a GoFiler pseudo presentation, not something that is exported.
XBRLGetPresentationProperties - Returns an array of properties about a specific presentation. It uses specific key - value pairs to identify properties. The available key values are:
| Key Name | Description | ||
| Name | Name of presentation. | ||
| RootElement | Root element of presentation. | ||
| TableElement | Table element for presentation. If blank, the default is used on export. | ||
| LineItemElement | Line Item (top abstract) element for presentation. If blank, the default is used on export. | ||
| DimensionElement-nnn | Dimension element for presentation, if applicable. Elements are numbered from 001 to 010. Empty elements are not added to the result. | ||
| DomainElement-nnn | Domain element for presentation, if applicable. Elements are numbered from 001 to 010. Empty elements are not added to the result. | ||
| RoleURI | Unique presentation role URI, if specified. |
The keys we want to pay attention to here are the DimensionElement-nnn and DomainElement-nnn ones. In GoFiler, a presentation has up to ten dimensions, so we’ll have to check DimensionElement-001 through DimensionElement010 to see which ones are defined. They should go up sequentially from 001 to 010, so once we find an empty one we can stop looking.
XBRLGetPresentationContexts - Gets an array of the context identifiers that are used in a presentation.
XBRLGetContext - Similar to the XBRLGetPresentationProperties function, it gets an array with defined key values that correspond to properties of the context. Here are the values defined for it.
| General Items: | ||||
| ID | The context’s ID. This can essentially be any legal XML/XBRL. | |||
| ContextIndex | The internal context ID. This is not an XBRL provided data element. If an index was used to retrieve the context data, this value will match that index. | |||
| CIK | EDGAR CIK value for the context. | |||
| StartDate | Starting date of the context. | |||
| EndDate | Ending date of the context. If this field is not present, the context is considered an instant context, otherwise it is considered a duration context. | |||
| Dimensional Items: | ||||
| MemberElement-nnn | Member element name (namespace and element). | |||
| MemberLabel-nnn | Member label. | |||
| DimensionElement-nnn | Dimension element name (namespace and element). | |||
| DimensionLabel-nnn | Dimension label. | |||
The “frame” of this script is copied from the previous post. Some modifications were made, however. The old run function is renamed to run_tests, and it just calls our validate functions. The contents of the previous run function were put into the new function run_custom_check, so our test for number of custom elements is still here. The function run_cycle_check is where all our new logic goes. I also added a debug_msg function, which is a type of function I use in a lot of the scripts I write. If you define your own log function, you can enable or disable all debug logging by defining a variable and just checking it in your debug function. In this case, if the script is running from the IDE, the debug_msg function prints out a message to the log. If it isn’t run in IDE mode, the function doesn’t do anything at all. This makes it really easy to debug your script without having the debug information effecting your final script.
Let’s take a look at our new function, and discuss how it works.
/****************************************/
void run_cycle_check(int f_id, string mode, handle window){ /* run cycle check */
/****************************************/
-- variable declarations omitted --
/* */
if (mode != "preprocess"){ /* if not running in preprocess mode */
return; /* return */
} /* */
if (IsWindowHandleValid(window) == false){ /* if the window isn't valid */
window = GetActiveEditWindow(); /* get the edit window */
wType = GetEditWindowType(window) & EDX_TYPE_ID_MASK; /* get the type of the window */
if (wType != EDX_TYPE_XBRL_VIEW){ /* if the type isn't xbrl view */
return; /* exit */
} /* */
hooked = true; /* store that we're running hooked mode */
} /* */
The function starts off with our normal check to make sure we’re running in preprocess mode, then checks to see if a window handle was passed. If no window handle was passed to the function (we are only going to get a handle if it’s run from the IDE) then we need to use the GetActiveEditWindow function to grab the handle of the current window. If it’s not XBRL, we can return, otherwise we can set hooked to true for future reference.
XBRL = XBRLGetObject(window); /* get the XBRL object */
presentations = XBRLGetPresentations(XBRL); /* get a list of all presentations */
all_contexts = XBRLGetContexts(XBRL); /* get a list of all contexts */
num_pres = ArrayGetAxisDepth(presentations); /* get the number of presentations */
for (ix = 0; ix < num_pres; ix++){ /* for each presentation */
Now we can actually start the XBRL processing. We can grab the XBRL object with XBRLGetObject. Then for the XBRL, get a list of all the presentations, get a list of all contexts, and get the number of presentations. Then we can iterate over each presentation.
ArrayClear(pres_dimension_props); /* clear array after scan */
ArrayClear(pres_props); /* clear array after scan */
if (XBRLGetPresentationType(XBRL,ix)<500){ /* if the presentation is not pseudo */
debug_msg(""); /* spacer */
debug_msg("Checking presentation: "+presentations[ix]); /* debug message */
pres_props = XBRLGetPresentationProperties(XBRL,ix); /* get properties of presentation */
pres_contexts = XBRLGetPresentationContexts(XBRL,ix); /* get contexts on presentation */
for (rx=1; rx<10; rx++){ /* for all possible dimensions */
The two ArrayClear’s here are needed to reset the arrays between presentations. I forgot them at first, leading to all sorts of false positives because dimensions from previous presentations were being left in the array. Then we can use XBRLGetPresentationType to check to see if this presentation is an actual XBRL presentation. If it is, we need to get the properties of the presentation, and get a list of all the contexts in the presentation. Next we need to check all the contexts in the presentation. We don’t have a way of getting the number of dimensions, so we can just set our for loop to run ten times, because ten is the maximum number of dimensions a presentation can use in GoFiler.
d_key = FormatString("DomainElement-0%02d",rx); /* test domain key */
a_key = FormatString("DimensionElement-0%02d",rx); /* test axis (dimension) key */
if (pres_props[d_key] == "" || pres_props[a_key] == ""){ /* if the keys have no valid values */
rx = 10; /* we're done counting */
} /* */
else{ /* else, they have valid values */
debug_msg(" axis : "+pres_props[a_key]); /* debug message */
debug_msg(" domain: "+pres_props[d_key]); /* domsin */
pres_dimension_props[rx-1][0] = pres_props[a_key]; /* store values of this key in table */
pres_dimension_props[rx-1][1] = pres_props[d_key]; /* store values of this key in table */
} /* */
} /* */
If you check the documentation for the XBRLGetPresentationProperties function, you’ll see that all of the domain names and dimension names are in the same format. So we need to create strings that will act as keys for each of these in our for loop, our variables d_key (domain key) and a_key (axis key). We can check if those keys are blank in our array, because if they are, it means we have no dimension to look at, and can set our counter to the end of the for loop to break out of it. If the keys have values, it means this is a valid dimension, so we can store it in our table of presentation dimension properties. I decided to use column zero of the table as my axis, and column one as my domain. Each row represents a different dimension.
num_dimension_props = ArrayGetAxisDepth(pres_dimension_props); /* get the number of actual dimensions */
if (num_dimension_props>0){ /* as long as we have dimensions */
pres_contexts = XBRLGetPresentationContexts(XBRL,ix); /* get the contexts on the presentation */
num_contexts = ArrayGetAxisDepth(pres_contexts); /* get the number of contexts */
for(rx=0; rx<num_contexts; rx++){ /* for each context in the presentation */
num_dimensions = 0; /* reset number of dimensions */
if (pres_contexts[rx]!=""){ /* if we have a context to look at */
c_id = FindInList(all_contexts,pres_contexts[rx]); /* find this context in the list */
context = XBRLGetContext(XBRL,c_id); /* get all context properties */
cm_key = "MemberElement-0%02d"; /* initialize value for member key */
cd_key = "DimensionElement-0%02d"; /* initialize value for domain key */
Now that we’ve built a table of dimension properties for our presentation, we can get the number of dimensions by checking the axis depth of our table. If there are dimensions, we need to do more checking, otherwise we can move to the next presentation. All we’ve done so far is build a list of dimensions in the presentation properties, now we need to build a list of all the dimensions used in the contexts of the presentation. Then we can check to see if we have any conflicts. First, we get the list of contexts in the presentation and the count of contexts, and loop over each of them. We can reset our number of dimensions counter back to zero, check to make sure this context isn’t empty, and then get our actual context ID from the list of all contexts. The XBRLGetPresentationContexts function returns the names of the contexts, not the actual ID, so we need that before we can do anything with the context. Once we have the ID, we can use XBRLGetContext to get the properties for the context, and create key names similar to what we did for the presentation properties.
/* * get num of used contexts */
for(bx=1; bx<10; bx++){ /* for each possible context in the list*/
if (context[FormatString(cm_key,bx)]!=""){ /* if the context isn't blank */
num_dimensions++; /* increment the number of used dims */
} /* */
} /* */
/* for each dimension in the context */
for(bx=0; bx<num_dimensions; bx++){ /* */
cm_key = FormatString(cm_key, bx+1); /* build the member key */
cd_key = FormatString(cd_key ,bx+1); /* build the domain key */
for(cx=0; cx<num_dimension_props; cx++){ /* for each context */
Next there’s a loop to check for how many non-blank contexts are in our presentation. For each one it finds, it just adds one to the number of dimensions. Then we can just set our next for loop to run for that number of times, and build new context member and context dimension keys for each dimension in the context we’re looking at. Then we just need to loop through each dimension in the context.
if ((pres_dimension_props[cx][1] == context[cm_key])||/* if the context member is a domain */
(pres_dimension_props[cx][1] == context[cd_key]) || /* or context dimension is a domain */
(pres_dimension_props[cx][0]==context[cm_key])){ /* or context member is an axis */
debug_msg("pres name = %s", /* debug messages */
presentations[ix]); /* * */
debug_msg("pres_domain = %s", /* * */
pres_dimension_props[cx][1]); /* * */
debug_msg("pres_dimension = %s", /* * */
pres_dimension_props[cx][0]); /* * */
debug_msg("context_dimension = %s", /* * */
context[cd_key]); /* * */
debug_msg("context_member = %s", /* * */
context[cm_key]); /* * */
debug_msg(" "); /* * */
bad_contexts[bc][0] = ix; /* store presentation name */
bad_contexts[bc][1] = c_id; /* store context id */
bc++; /* increment number of bad contexts */
cx = num_dimension_props; /* set value to end for loop */
bx = num_dimensions; /* set value to end for loop */
For each dimension in our context, if the context’s member equals our presentation’s domain, or the context’s dimension equals the presentation’s domain, or the context’s member equals the presentation’s axis, we know we have a problem. I have a bunch of debug messages in here I left, but really all we need to do at this point is add it to our bad_contexts table, increment the number of bad contexts, and set our counter variables cx and bx to the end values of their respective loops, to stop looping through our dimensions.
} /* */
} /* */
} /* */
} /* */
} /* */
} /* */
} /* */
} /* */
log = LogCreate("Check Dimension Elements"); /* create log */
AddMessage(log,"Checking Dimension Elements in %s", /* add start message to log */
GetEditWindowFilename(window)); /* * */
num_bad = ArrayGetAxisDepth(bad_contexts); /* get the number of bad contexts */
if (num_bad>0){ /* if there are any */
MessageBox('x',CYC_ERROR); /* display error message */
AddMessage(log,CYC_ERROR_TAB); /* add message to the log */
for(ix = 0; ix < num_bad; ix ++){ /* for each error */
if (id!=bad_contexts[ix][0]){ /* if first time showing this pres */
id = bad_contexts[ix][0]; /* remember context was shown */
AddMessage(log," %s",presentations[id]); /* display presentation name */
} /* */
c_id = bad_contexts[ix][1]; /* get ID of bad context */
AddMessage(log," %s",all_contexts[c_id] ); /* add bad context name to log */
} /* */
} /* */
else{ /* */
AddMessage(log," No errors found"); /* display no errors message */
} /* */
LogDisplay(log); /* display the log */
} /* */
That should be enough to get us through our entire XBRL report, all presentations, to build our bad_contexts table. Now we just need to display that to the user in a somewhat intelligent manner. We can create a log, and if we have any bad contexts, display an error message and for each bad presentation in our table we want to print out each context that actually has a problem. Then we can display the log to the user, and end our function.
Here's a finished script, containing our new function, with modifications to allow both our new and old tests to run on validate.
#define HIGH 30
#define MID 15
#define HIGH_MSG "Using a high percentage of extended elements is not recommended. Use more taxonomy elements."
#define MID_MSG "Custom element usage is about average, if possible replace some custom definitions for taxonomy elements."
#define CYC_ERROR "Found cyclic presentation relationships. See the 'Check Presentation Elements' tab for more information."
#define CYC_ERROR_TAB "Found cyclic presentation relationships. This means you have a dimension or member in a context that is used as a parent of itself in the presentation properties."
void setup();
void run_custom_check(int f_id, string mode, handle window);
void run_cycle_check(int f_id, string mode, handle window);
void run_tests(int f_id, string mode);
void debug_msg(string msg);
boolean debug;
void main(){
int ix;
int size;
string windows[][];
handle window;
if (GetScriptParent() == "LegatoIDE"){
debug = true;
windows = EnumerateEditWindows();
size = ArrayGetAxisDepth(windows);
for (ix = 0 ; ix < size; ix++){
if (windows[ix]["FileTypeToken"] == "FT_XFR"){
run_cycle_check (0,"preprocess",MakeHandle(windows[ix]["ClientHandle"]));
}
}
}
setup();
}
void debug_msg(string msg){
if (debug == true){
AddMessage(msg);
}
}
void setup(){
MenuSetHook("XBRL_VALIDATE", GetScriptFilename(), "run_tests");
MenuSetHook("EDGAR_VALIDATE", GetScriptFilename(), "run_tests");
}
/****************************************/
void run_tests(int f_id, string mode){ /* run all tests */
/****************************************/
run_cycle_check(f_id, mode, NULL_HANDLE); /* run the cycle relation check */
run_custom_check(f_id, mode, NULL_HANDLE); /* check for custom elements */
} /* */
/****************************************/
void run_cycle_check(int f_id, string mode, handle window){ /* run cycle check */
/****************************************/
string pres_props[]; /* presentation properties */
int id,c_id; /* bad presentation id's, context id's */
handle log; /* log file */
int bad_contexts[][]; /* bad contexts */
handle XBRL; /* XBRL object */
string pres_dimension_props[][]; /* properties of the presentation dims */
string presentations[]; /* list of all presentations */
string pres_contexts[]; /* list of contexts in a presentation */
string all_contexts[]; /* summary of all contexts */
string context[]; /* properties of a single context */
string a_key; /* axis key */
string d_key; /* dimension key */
string cd_key; /* context dimension key */
string cm_key; /* context member key */
dword wType; /* window type */
int bc; /* bad context counter */
int ix, rx, bx, cx; /* iterator variables */
int num_contexts; /* number of contexts */
int num_pres; /* number of presentations */
int num_bad; /* number of bad contexts */
int num_dimensions; /* number of dimensions */
int num_dimension_props; /* number of dimension props in pres */
boolean hooked; /* if running in hooked mode or not */
/* */
if (mode != "preprocess"){ /* if not running in preprocess mode */
return; /* return */
} /* */
if (IsWindowHandleValid(window) == false){ /* if the window isn't valid */
window = GetActiveEditWindow(); /* get the edit window */
wType = GetEditWindowType(window) & EDX_TYPE_ID_MASK; /* get the type of the window */
if (wType != EDX_TYPE_XBRL_VIEW){ /* if the type isn't xbrl view */
return; /* exit */
} /* */
hooked = true; /* store that we're running hooked mode */
} /* */
XBRL = XBRLGetObject(window); /* get the XBRL object */
presentations = XBRLGetPresentations(XBRL); /* get a list of all presentations */
all_contexts = XBRLGetContexts(XBRL); /* get a list of all contexts */
num_pres = ArrayGetAxisDepth(presentations); /* get the number of presentations */
for (ix = 0; ix < num_pres; ix++){ /* for each presentation */
ArrayClear(pres_dimension_props); /* clear array after scan */
ArrayClear(pres_props); /* clear array after scan */
if (XBRLGetPresentationType(XBRL,ix)<500){ /* if the presentation is not pseudo */
debug_msg(""); /* spacer */
debug_msg("Checking presentation: "+presentations[ix]); /* debug message */
pres_props = XBRLGetPresentationProperties(XBRL,ix); /* get properties of presentation */
pres_contexts = XBRLGetPresentationContexts(XBRL,ix); /* get contexts on presentation */
for (rx=1; rx<10; rx++){ /* for all possible dimensions */
d_key = FormatString("DomainElement-0%02d",rx); /* test domain key */
a_key = FormatString("DimensionElement-0%02d",rx); /* test axis (dimension) key */
if (pres_props[d_key] == "" || pres_props[a_key] == ""){ /* if the keys have no valid values */
rx = 10; /* we're done counting */
} /* */
else{ /* else, they have valid values */
debug_msg(" axis : "+pres_props[a_key]); /* debug message */
debug_msg(" domain: "+pres_props[d_key]); /* domsin */
pres_dimension_props[rx-1][0] = pres_props[a_key]; /* store values of this key in table */
pres_dimension_props[rx-1][1] = pres_props[d_key]; /* store values of this key in table */
} /* */
} /* */
num_dimension_props = ArrayGetAxisDepth(pres_dimension_props); /* get the number of actual dimensions */
if (num_dimension_props>0){ /* as long as we have dimensions */
pres_contexts = XBRLGetPresentationContexts(XBRL,ix); /* get the contexts on the presentation */
num_contexts = ArrayGetAxisDepth(pres_contexts); /* get the number of contexts */
for(rx=0; rx<num_contexts; rx++){ /* for each context in the presentation */
num_dimensions = 0; /* reset number of dimensions */
if (pres_contexts[rx]!=""){ /* if we have a context to look at */
c_id = FindInList(all_contexts,pres_contexts[rx]); /* find this context in the list */
context = XBRLGetContext(XBRL,c_id); /* get all context properties */
cm_key = "MemberElement-0%02d"; /* initialize value for member key */
cd_key = "DimensionElement-0%02d"; /* initialize value for domain key */
/* * get num of used contexts */
for(bx=1; bx<10; bx++){ /* for each possible context in the list*/
if (context[FormatString(cm_key,bx)]!=""){ /* if the context isn't blank */
num_dimensions++; /* increment the number of used dims */
} /* */
} /* */
/* for each dimension in the context */
for(bx=0; bx<num_dimensions; bx++){ /* */
cm_key = FormatString(cm_key, bx+1); /* build the member key */
cd_key = FormatString(cd_key ,bx+1); /* build the domain key */
for(cx=0; cx<num_dimension_props; cx++){ /* for each context */
if ((pres_dimension_props[cx][1] == context[cm_key])||/* if the context member is a domain */
(pres_dimension_props[cx][1] == context[cd_key]) || /* or context dimension is a domain */
(pres_dimension_props[cx][0]==context[cm_key])){ /* or context member is an axis */
debug_msg("pres name = %s", /* debug messages */
presentations[ix]); /* * */
debug_msg("pres_domain = %s", /* * */
pres_dimension_props[cx][1]); /* * */
debug_msg("pres_dimension = %s", /* * */
pres_dimension_props[cx][0]); /* * */
debug_msg("context_dimension = %s", /* * */
context[cd_key]); /* * */
debug_msg("context_member = %s", /* * */
context[cm_key]); /* * */
debug_msg(" "); /* * */
bad_contexts[bc][0] = ix; /* store presentation name */
bad_contexts[bc][1] = c_id; /* store context id */
bc++; /* increment number of bad contexts */
cx = num_dimension_props; /* set value to end for loop */
bx = num_dimensions; /* set value to end for loop */
} /* */
} /* */
} /* */
} /* */
} /* */
} /* */
} /* */
} /* */
log = LogCreate("Check Dimension Elements"); /* create log */
AddMessage(log,"Checking Dimension Elements in %s", /* add start message to log */
GetEditWindowFilename(window)); /* * */
num_bad = ArrayGetAxisDepth(bad_contexts); /* get the number of bad contexts */
if (num_bad>0){ /* if there are any */
MessageBox('x',CYC_ERROR); /* display error message */
AddMessage(log,CYC_ERROR_TAB); /* add message to the log */
for(ix = 0; ix < num_bad; ix ++){ /* for each error */
if (id!=bad_contexts[ix][0]){ /* if first time showing this pres */
id = bad_contexts[ix][0]; /* remember context was shown */
AddMessage(log," %s",presentations[id]); /* display presentation name */
} /* */
c_id = bad_contexts[ix][1]; /* get ID of bad context */
AddMessage(log," %s",all_contexts[c_id] ); /* add bad context name to log */
} /* */
} /* */
else{ /* */
AddMessage(log," No errors found"); /* display no errors message */
} /* */
LogDisplay(log); /* display the log */
} /* */
void run_custom_check(int f_id, string mode, handle window){
handle XBRL;
string presentations[];
string elements[];
dword wType;
int fields_pos;
int customs;
int total_elements;
int ix, rx;
int percent;
int size;
boolean hooked;
if (mode != "preprocess"){
return;
}
if (IsWindowHandleValid(window) == false){
window = GetActiveEditWindow();
wType = GetEditWindowType(window) & EDX_TYPE_ID_MASK;
if (wType != EDX_TYPE_XBRL_VIEW){
return;
}
hooked = true;
}
XBRL = XBRLGetObject(window);
presentations = XBRLGetPresentations(XBRL);
fields_pos = FindInList(presentations,"XBRL Financial Fields");
if (fields_pos < 0){
return;
}
elements = XBRLGetPresentationElements(XBRL,fields_pos);
size = ArrayGetAxisDepth(elements);
customs = 0;
for (ix = 0; ix < size; ix++){
if (elements[ix] !=""){
total_elements++;
if (FindInString(elements[ix],"custom:")>(-1)){
customs++;
}
}
}
percent = (customs*100)/total_elements;
if (hooked == false){
AddMessage("Checking Custom Elements in %s",GetEditWindowFilename(window));
AddMessage("Found %d total line items",total_elements);
AddMessage("Found %d custom line items",customs);
AddMessage("%d%% custom line items",percent);
}
else{
if (percent > MID){
if (percent >= HIGH){
MessageBox('x',"%d%% custom elements used as line items. %s",percent,HIGH_MSG);
}
else{
MessageBox('i',"%d%% custom elements used as line items. %s",percent,MID_MSG);
}
}
}
}
Working with the XBRL Object to get information can be a bit trickier, because sometimes you’ll need to go through a few steps to get identifiers. These identifiers can be used to get the actual information you’re looking for, allowing you to accomplish your original goal. Once you know how the data is stored, it makes sense, and you can design your script to get the pieces you need. This script is easily extensible with more validations, all you’d have to do to write a new validator is add a new function, and then add add the function call into the run_tests function. Easy.
 Steven Horowitz has been working for Novaworks for over five years as a technical expert with a focus on EDGAR HTML and XBRL. Since the creation of the Legato language in 2015, Steven has been developing scripts to improve the GoFiler user experience. He is currently working toward a Bachelor of Sciences in Software Engineering at RIT and MCC. |
Additional Resources
Legato Script Developers LinkedIn Group
Primer: An Introduction to Legato
Quicksearch
Categories
Calendar
Feeds
