FixSecurityBug Luthier design (#914)
* Base design
* Refactors to base projects
* Lua generating code
* Java skeleton
* Updates design.
-- Factories are now registered through the `LuthierIndustrialPark`
builder method. The `LuthierIndustrialPark` relies on instances of
`FactoryPark<T>` to handle extracting config and invoking the
appropriate factory.
* [updated] fixed list rendering in documentation (#917)
* Various fixes Centering the dimensionConfig issues (#916)
* ignored the external config jsons, encourage the rebuild of these instead of providing a copy from the repo right away. \nWill also make the tutorial much clearer eventually.
* Fixing Various Issues regarding JSON formatting
--- config.lua : added script to mkdir
--- dimensions.lua : makes sure the spelling is correct
--- in particular desc -> description to keep consistent
--- lib/json.lua : tried to format output lua to a prettier
format, not quite there. Recommend using an external
formatting tool in near future
--- utils/dimensionUtils.lua : corrected the top level
JSON to be a look-up table instead of a list.
* Addressed a couple of comments on the last pr
--- Reverted back to non-formatted json.lua
--- removed unnecessary spaces
* Fixes typo in `LuthierIndustrialPark`
-- A typo meant that `getPhysicalTableDictionary()` was invoking itself,
rather than delegating to `resourceDictionaries`. That has been fixed.
* Implements KeyValueStoreDimensionFactory (#919)
While implementing the KeyValueStoreDimensionFactory, we also made the following changes:
-- Fixes some slightly incorrect documentation in the Lua config.
* resolved the default value of
-- defaultFields
-- isAggregatable
-- description
Also added in appropriate tests.
-- Includes the list of fields we know about for a dimension in the
`unknownDefaultField` error message. Keep in mind that customers may be
generating their dimension fields (or anything really) programatically,
so if we can include our view of the world (well, the pertinent bit) in
the error message, it'll go a long way to helping customers fix their
config.
-- Adds validation for every dimension field. Before, if a customer
forgot a field that we relied on, they'd get a *very* not helpful
`NullPointerException`. Now, we tell them which field is missing in
which dimension. Calling this method is before every field request in
every builtin factory will be less than ideal, and will become
unnecessary if/when we do the bean validation thing that @mclawhor
recommended, but that isn't here yet, and the config should be useable
even if we don't have time to do add bean validation anytime soon
(which is likely).
* A lua script that generates searchProviderTemplates (#924)
* Added appropriate testApplicationConfig (and moduleConfig as an example, which is overrode) to fix the dependency issue in SystemConfig (+9 squashed commits)
[1531bd19] fix string mismatch for luthier index path
[6ed9b347] fixed defaulting Optional in LuthierBinderFactory in the ConfigurationLoader
[520bf9e8] another try at setting up a Lucene SP that Travis can build
[7b41ac62] integrated SearchProvider to the resource dictionaries;
cleaned up a couple of unused imports in tests
[e498c2ef] fixed up some tests:
More meaningful Luthier Industrial Park tests
[0309df7f] Style Fixes
[3038eacd] rewrote the logic of searchProvider Handling, now tests are also more enriched.
[6d0f58f7] fixed a merge conflict that I did not see
[44f0aab4] Rebase finished: searchProviderTemplate (+16 squashed commits)
Squashed commits:
[6d40e02e] finished cleaning up, addressed all comments.
[a6035153] partially solved the issues.
[b382fad0] Spent an amazing(ly long) time to debug the groovy syntax, but we are done now
[5933a553] renamed all luthier *Test -> *Spec
[03da2de4] removed apiName entry from DimensionConfig.json
[26d6eb0a] now passing regression test, cleaned up structure. TODO:
-- Add hints for missing json entries
-- Build test on an instantiated factory
[2e765b14] structure implemented, to be completed
[fa491dff] Two changes:
-- refactored the DefaultDimensionFactory to KeyValueStoreDimensionFactory
-- various style fixes
[7b0ca3c8] modified the .gitignore file so now CI can find the test resource (+1 squashed commit)
Squashed commits:
[f3a7643d] amending the previous commit
[429a7528] added additional test case to check key existence for dimensionConfig.json;
also fixed the error in industrialPark that caused trouble with the nested Builder class;
built a sample default DimensionFactory to pass the load test in LuthierIndustrial Park.
[fb44f0a3] currently the load() triggers a NullPointerException in Factory.buildEntity
[75796095] a lot of small changes in styles, typos, and comments. Detailed description in PR #919 follow-up discussions
[b5fb4075] finished testing for ResourceNodeSupplier; fixed various small typo along the way
[0249864e] temp saving the works on factories
[38d99678] halfway into building the test
[fe8ad912] Various fixes Centering the dimensionConfig issues (#916)
* ignored the external config jsons, encourage the rebuild of these instead of providing a copy from the repo right away. \nWill also make the tutorial much clearer eventually.
* Fixing Various Issues regarding JSON formatting
--- config.lua : added script to mkdir
--- dimensions.lua : makes sure the spelling is correct
--- in particular desc -> description to keep consistent
--- lib/json.lua : tried to format output lua to a prettier
format, not quite there. Recommend using an external
formatting tool in near future
--- utils/dimensionUtils.lua : corrected the top level
JSON to be a look-up table instead of a list.
* Addressed a couple of comments on the last pr
--- Reverted back to non-formatted json.lua
--- removed unnecessary spaces (+1 squashed commit)
Squashed commits:
[8b14a6b4] Lua generating code
* SearchProviderTemplate implementation
* fixed typo in commit, groovy style fixes
* [patch] reverted the package name in luthier testApplication.properties to 'bard' (#929)
* Redid the lua code
- generate more useful fields to build Physical Table.
- comments vastly improved.
- split functions to logical build and physical build.
- json files also up to date.
* PhysicalTable support on the FactoryPark and LuthierIndustrialPark side
* clean up validation code style and other small fixes
* factories to support physicalTables of single data source
* Data structures to support the build of physicalTables
* Test on whether a physicalTable is loaded with correct information, as specified in its json.
* Key value store using factory to build (#936)
* lua generation for KeyValueStore related config
* keyValueStore now uses a factory-dictionary style initialization and loading.
* Luthier Logical Table Configuration (#938)
* Lua generation script for LogicalTable
* Granularity dictionary is build in LIP to support table building
* java side support for LogicalTableGroup
* Tests and JSON config files up to date with the LogicalTableGroup.
* Lua defaulting (#940)
* has a version control (snapshots) of each json the verify_then_config.lua build (+10 squashed commits)
Squashed commits:
[aa5bd48c] logical table and physical table defaulting done, also making sure that the json files (including ref files) are updated.
[e101c2c7] removed unused utilities.lua
[e1ef7d30] makes sure all error messages are well formatted, using tostring()
[bffbfd21] Removed unnecessary printouts
[9958275f] new lua util to compare correctness between files.
[9c6b3304] defaulting strategy for dimensions.
[1ae8a297] fix javadoc.
[8b29f02c] LogicalTable tested using transitive testing against physicalTable it depends on. Also updated all json files.
[ef513be0] LogicalTable implemented, using the LogicalTableGroup approach
[ba5e113b] changed design, now we build a LogicalTableGroup from factory.
Also passing regression tests. (+20 squashed commits)
Squashed commits:
[be7a29a9] save
[1472940c] intermediate save
[c75e043e] intermediate, need fix: logical table reading method
[c2d851cc] logical table implementation finished, including structures.
desperately needs testing
[61d505ae] lifted the nested bean class to data.config
[9e0bb882] Cleaned up and documented the generation of physicalTable
[15ee8c02] new addition to test,
Making it ready for logicalTable to branch off from here
[cba9b845] Address more suggestions on PR, clean up style
[417b289f] Swapping to ENTITY_NAME when referring to the const string used to call validation util
[6c3f5d0f] Style clean up and java doc addition
[2ecd29a4] suggested changes to PR
[90b752c9] - New test on a specific Dimension
- Code quality fixes suggested by Codacy
[1da2ddd9] Styles fixes, local build passes
[481a7995] Clean up commented code I forgot to delete
[ab6a5a7d] Implemented the strict physical table with minimum test
[826b75bd] Lua generation of table separated into physical and logical two files
[d3575b07] Stream-lined the design on the hierarchy of physicalTable's factories with one truth source.
I quite like the refactor on logicalToPhysicalColumnNames, needs a hell lotta testing, though.
The other arguments also looks promising. They will be implemented next.
[028a18e6] initial works on PhysicalTable
[7108c4f4] lua code now generate a default list of physicalTables (for both logical and physical tables) to help us decide whether it is a composite physical table or a base physical table
[52f4c283] starting point
* example outcome of using the verify_then_config
* Logical Table defaulting also up to date.
* removed unused files
* change default of dimension description from name -> longName
* Luthier metric maker stuff (#942)
* Finished rebase (+2 squashed commits)
Squashed commits:
[fd161d9b] Cleanup
[613c61f4] Not quite working impl of metric makers
* Reduced duplication.
* Hacked out a working MetricMakerUtil and now metric maker can be built into json, albeit no defaulting, exact. (#944)
* Sample field documentation. (#937)
* Sample field documentation.
-- This commit provides an example of what the configuration
documentation might look like.
-- My goal is to make the documentation clear, skimmable, and
searchable, both visually and with tools. The format is as follows:
There are two sections: `Required Fields` and `Optional Fields`. Each
section is a list.
Each entry in the Required Field list has the following format:
```
<li> {@code fieldName} - Type. Brief description of the field's purpose.
```
Each entry in the Optional Field list has the following format:
```
<li> {@code fieldName} - Type. Default: {@code value}. Brief description of the field's purpose.
```
Ideally, the whole thing should fit on one 120 character line, but can
be wrapped if a clear description cannot be condensed into 120ish
characters. If a field is a collection of ObjectNodes, then the fields
on its ObjectNode should be defined in a nested list that follows the
same format.
* Fixed style errors.
* Fixes a few typos in the metrics.lua code. (#947)
-- Updates some slightly wrong documentation.
-- We were using the wrong names for the average aggregation makers in
metrics.lua.
* Finalizing Luthier Metric Maker and Metric (#946)
* long sum maker in place and build correctly. Will proceed to connect this to an actual logicalMetric
* AggregationAverageMaker in place and tested. This is the class that builds all sort of dailyAvg/monthlyAvg, etc.
* MetricConfig build now consistent with the defaulting strategy
* checkpoint: a longSum Metric builds
* Made sure that the tests pass. The design of AggregationMetric and PostAggregationMetric reflected.
* Addressed comments, doc fixes mostly
* Discussions addressed on PR.
* DoubleSum Maker
* operation -> function; dataType no longer required; json updated.
* CountMakerFactory
* metrics broken down into aggregation_metrics and post_aggregation_metrics. Then some of the configs can be simplified.
* ignores snapshots
* finished everything before the average metric's implementation
* Removed unused imports
* Revised some field validation function calls, beautified them
* Wikipedia example fully working
* new way to repopulate LIP, import sequence fix, DefaultLogicalTableGroupFactory partially done
* improved the LuthierValidationUtil to support multiple verification directly.
* catch exceptions more gracefully.
* refactored away an awkward naming
* default the LogicalMetric and LogicalTable types.
* Luthier wiki (#949)
* temp stuff for webservice
* Merge branch 'metricMakerFinalize' of github.com:gab-umich/fili into metricMakerFinalize
Build passes.
* 'metricMakerFinalize' of github.com:gab-umich/fili:
Fixes a few typos in the metrics.lua code. (#947)
* Working luthier demo
* beautified configs, cleaned up unused/commented code in LIP
* nice bash hint added
* addressed a few comments.
* changed place where we update the metadata service
* removed unused imports
* changed location, added comments on usage
* now supports StorageStrategy, name change on DimensionConfig's "domain" -> "dimensionDomain"
* fix line length to pass style checking
* Organize luthier (#951)
* Lua done
* updated helper script and added readme
* clean ups on the new documentation, removed unused script
* Fixed a non-agg dimension in table that should be aggregatable
Fixed script to work correctly with respect to directory.
* Built the command line tool to put jsons from resources/ to target/classes by default, and other places on demand.
* resolved a hidden bug when calling the camelCase() naively without checking (when table name is already camelCased)
* fixed a couple of style issues.
* removed test resources, make the test part of mvn test cycle
* Update organize (#8)
* Reduce redundant scripts and provide some default arguments.
* fixed comment
* Didn't invoke the right project.
* used <execution> specific <configuration> in pom.xml, must supply exec:java@<id> to target it
* fixed defaulting syntax and style to make script more robust.
* only run tests if Lua is installed to /usr/local/bin/lua
* mvn correctly skips tests if lua not detected.
* let Travis install Lua
* travis rerun
* reworked README.md
* rerun travis, again
* fix up doc, travis rerun
* addressed code smells
* Address codacy stuff (#953)
* Respond to a number of codacy complaints.
More tests.
Exclude the main class from code coverage.
Fix some reported code smells.
* Try to ignore the value object security complaint.
Moved test file locations.
* More warning cleanup
* Clarified comment
* Excluding the lombok class
