Auditing & Data History
- Candice Louw (Unlicensed)
Auditing
To comply with data protection regulation, we need to make sure that SORMAS provides an audit log trail which can be easily ingested by dedicated log processing systems and allows investigation by officials.
Use cases:
User opens case in UI -> call to backend method CaseFacade.getCaseDataByUuid needs to be logged
User edits/deletes case in UI -> call to backend method CaseFacade.save/deleteCase needs to be logged
User does export -> call to CaseFacade.getExportList needs to be logged
User opens case directory -> call to CaseFacade.getIndexList needs to be logged
High-Level Explanation
The audit trail gets populated by automatically logging every invocation of a facade/EJB method. By this, we can trace every interaction with the system (i.e., via Vaadin UI or REST). We output the collected logs to a user configurable log sink such that the logs can be easily ingested for further processing.
The most important module that is covered is the SORMAS backend.
Related epic: https://github.com/hzi-braunschweig/SORMAS-Project/issues/7904
Setup
Audit logging can be set up by setting the audit.logger.config property in the sormas.properties file to a path that points to a logback configuration file.
There is an example file in SORMAS source code in the sormas-base/setup folder. This writes audit logs to a file but Logback can be easily configured to write to other destinations like a database or a log processing system.
For example adding a Loki4jAppender appender to the logback configuration file will send the logs to a Loki instance.
e.g. The following configuration will send the logs to a Loki instance running on localhost:
<configuration>
<appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
<http>
<url>http://localhost:3100/loki/api/v1/push</url>
</http>
<format>
<label>
<pattern>app=my-app,host=localhost,level=%level</pattern>
</label>
<message>
<pattern>l=%level h=localhost c=%logger{20} t=%thread | %msg %ex</pattern>
</message>
<sortByTime>true</sortByTime>
</format>
</appender>
<root level="DEBUG">
<appender-ref ref="LOKI"/>
</root>
</configuration>NOTE: in order to make the Loki appender work, you need to add the latest
loki-logback-appender.jaras a dependency in the libs folder of the SORMAS domain in your payara
Log Format
For the general purpose of audit logging SORMAS logs access to the SORMAS backend with details on what exactly has been accessed, without including sensitive data (e.g. names).
For the logging output format, SORMAS is using the FHIR R4 AuditEvent resource, which is based on the IHE ATNA audit profile. FHIR is a well-known and established standard for exchange of medical related data.
A FHIR AuditEvent resource has a specified JSON representation which is compact and easy to digest by log processing systems like Loki or ELK.
These are the most important fields logged in the AuditEvent class:
Content | Description | Field in |
|---|---|---|
Access type | Create/Read/Update/Delete/Execute |
|
timestamp | Timestamp of the event |
|
actor | Functional instance causing the event (e.g., user). Users should be identified with a human readable name besides using the UUID |
|
Executing instance | Identifier of the system(component) generating the audit event |
|
Activity/Event | Description of the activity with unique reference(uuid) w.r.t. the accesses data |
|
Additionally, there is the field type, which will be populated according to the following table.
Code | System | Name | Description |
|---|---|---|---|
110100 | Application Activity | Start, Stop | |
110106 | Export | Database level export | |
110112 | Query | A request for multiple entities | |
110110 | Patient Record | Use for read/write/delete operations on patient related entities (case, contact, symptoms, samples, ...) | |
object | An Operation on other Objects | Use for read/write/delete operations on all other entities, most importantly users, infrastructure, configuration and tasks |
Logged events
SORMAS logs the following events:
Application lifecycle events (start, stop)
Data reads
Creation and change of data
Deletion of data
REST api calls
Failed login attempts
External message adapters can also log events using the AuditLoggerFacade
Logged data
Under all circumstances the principle of data minimization is followed
Only the UUID of entities is logged, not the full entity
The timestamps are in ISO format.
In general the log contains only pseudonymized personal data, the only exception being the name of the active user.
Log examples
Failed login
{ "resourceType": "AuditEvent", "type": { "system": "https://hl7.org/fhir/R4/valueset-audit-event-type.html", "code": "110114", "display": "User Authentication" }, "subtype": [ { "system": "https://hl7.org/fhir/R4/valueset-audit-event-sub-type.html", "code": "110122", "display": "Login" } ], "action": "E", "recorded": "2024-03-07T12:38:17.803+02:00", "outcome": "4", "outcomeDesc": "Authentication failed", "agent": [ { "name": "Username cannot be determined without ID token. Check Keycloak logs for details." } ], "source": { "site": "sormas.lu - UI MultiAuthenticationMechanism" }, "entity": [ { "what": { "reference": "sormas-ui/Callback" } } ] }Load case directory
{ "resourceType": "AuditEvent", "action": "R", "period": { "start": "2024-03-07T12:38:34+02:00", "end": "2024-03-07T12:38:34+02:00" }, "recorded": "2024-03-07T12:38:34.912+02:00", "outcomeDesc": "[CaseIndexDto(uuid=SYQMFJ-N2YXQ7-PMHXUE-SAQU2FIE),CaseIndexDto(uuid=RAG4WE-2BUPEN-3OXQW5-TB7OKE4Y),CaseIndexDto(uuid=WILIOE-FFX3L7-C46FKD-PCN6CJQU),CaseIndexDto(uuid=SCYB5H-MLZJJB-PFAVRD-6IQQ2FMU),CaseIndexDto(uuid=S7DQVF-4XCHEO-YA57KV-HHAHKF5Q),CaseIndexDto(uuid=XFLEBJ-A5FAY7-DGVLX7-XZ4VKP6U),CaseIndexDto(uuid=TUWWHZ-E5LQPN-GCVYDQ-VYTVKMAY),CaseIndexDto(uuid=RH7FHT-BMD6BV-T7V7VY-LIC4CHCM),CaseIndexDto(uuid=Q2YYLE-TXYILI-X5Q7FE-BJHL2IG4),CaseIndexDto(uuid=SF2Y72-YYG335-XI3ZFX-VTMBSMT4),CaseIndexDto(uuid=VORYUS-BB3TN5-EWQ6CB-TLM7KI4E),CaseIndexDto(uuid=UI47KF-UCZMA4-DHEHXU-MLMKKE5A),CaseIndexDto(uuid=TCO4DA-2GI64Y-D7IK2G-KBKACNXM),CaseIndexDto(uuid=T62OHY-7ID2Z2-LTVWSR-VCZVKMQE),CaseIndexDto(uuid=VCGLLU-REDG56-BVEULC-WZKASOOU),CaseIndexDto(uuid=T4ZZBB-2UHGS2-2TLMTH-26FOSBPQ),CaseIndexDto(uuid=SAGWGH-JS2LQA-WOV3F5-VAQICK5Y),CaseIndexDto(uuid=VZ4SMM-OBWY72-7DVLC4-6FZL2LFY),CaseIndexDto(uuid=V3ND6S-66VQIV-S642FO-5F42KIQ4),CaseIndexDto(uuid=VQFIG5-QDTIK7-UC2UAX-KD52CAHI),CaseIndexDto(uuid=QYL2XE-KU6BKX-CO7JXV-35PN2F4E),CaseIndexDto(uuid=VT3ADI-TYG6PJ-YYOHCT-UVWUKMPQ),CaseIndexDto(uuid=RVXCET-Y35DWT-3DYA34-4MXX2HLI),CaseIndexDto(uuid=RKU56K-HGXFNO-AFXOCX-YWH5KJZQ),CaseIndexDto(uuid=XABT2R-MVHQA4-PN3BNS-DWKKCCY4),CaseIndexDto(uuid=TDPU3J-ZSKZBD-BLRHEV-7PC2KELY),CaseIndexDto(uuid=X2TFPZ-BULBSB-7NVZIP-4K3D2ICA),CaseIndexDto(uuid=WKWEG3-EGYL2J-GKUHDC-F22CSBRU),CaseIndexDto(uuid=TOAQVR-IU3CKF-LXYJKL-IT6VKKOI),CaseIndexDto(uuid=QR2PLT-JVRN5H-BIMWJO-H3MJKAN4),CaseIndexDto(uuid=QS3UDV-QFVBMM-LGDB2F-LNCLSIRI),CaseIndexDto(uuid=WLJVI5-GNOLVN-D335JX-Q3HBKE7I),CaseIndexDto(uuid=UJLPOL-LSJ7ZS-MUWCCO-3FGQCICI),CaseIndexDto(uuid=XMXHHD-NE526P-EMUVCU-NGPDCFQ4),CaseIndexDto(uuid=UNPGCP-7HRLLF-JCRCZ6-HMAG2E6E),CaseIndexDto(uuid=Q44DQL-ECYSWP-XHRB5G-FNHXKEOA),CaseIndexDto(uuid=XZN26C-NZJXPS-3BLLPD-GDSD2GIQ),CaseIndexDto(uuid=RPZS4S-MIWQY6-7HZ6A5-SG7T2NRY),CaseIndexDto(uuid=RGJQ3S-2OBPHZ-AJY7CM-C6M3CNWU),CaseIndexDto(uuid=RXZ3FB-XNNB2M-TJZX2I-4C3EKAXI),CaseIndexDto(uuid=STPE5C-JV62ST-P7NTB4-5JMZCOQE),CaseIndexDto(uuid=WTP4W2-3DP76T-N4E5EV-YZESSKXM),CaseIndexDto(uuid=QUNZWX-AAKOXA-PEHSCB-DU6HKHSI),CaseIndexDto(uuid=WJOFHO-2X5NIA-VKPJGT-FON52IMU),CaseIndexDto(uuid=QGHGY4-B3RUOC-TX27YD-JUSQ2DPI),CaseIndexDto(uuid=RT4XRJ-VM74IK-5TYURK-B5NGKO2Y),CaseIndexDto(uuid=V6XOJS-SX4ESD-XQQSCS-JJXOCBQA),CaseIndexDto(uuid=RPVLIR-R7YK3E-QAXKU3-QVRFKLZA),CaseIndexDto(uuid=XSQTVG-4IXW7A-LAHYTW-2D5CKMDY),CaseIndexDto(uuid=W7UK6H-KZSPPD-PGHWIO-JI4MCCGU),CaseIndexDto(uuid=UMVX63-P6R4I3-SI7IK5-ZNEMSLC4),CaseIndexDto(uuid=VZKFEJ-Q5V2XU-5SQPRM-ZZACSDMQ),CaseIndexDto(uuid=RGH5HS-A3TSNU-5LZXIR-OOCICME4),CaseIndexDto(uuid=SYXQBI-UOXGW7-KPNDZU-ZCJNSDYU),CaseIndexDto(uuid=TJL5UI-QFCEOX-4NA4WA-UTVRCPJQ),CaseIndexDto(uuid=RDEGZP-OEFQZK-E3ZC5C-MYG2SHWY),CaseIndexDto(uuid=Q4ICNF-Y5R56L-NPSAXN-O52WCOUU),CaseIndexDto(uuid=USASG5-3HG5QV-GUZBS6-VKLOKF3A),CaseIndexDto(uuid=SRYNG6-7BL73Q-LAC6TK-V2H62NKI),CaseIndexDto(uuid=V6WVV4-MFF6G7-CBTDMQ-VNX7KG34),CaseIndexDto(uuid=UEQ4TR-QXWOXK-IERBBB-LE5SSNEA),CaseIndexDto(uuid=U2W5EI-34BFKM-FMCGON-4COT2F3M),CaseIndexDto(uuid=UFKBX6-WLAQRE-DOJNWN-2AFWSEYA),CaseIndexDto(uuid=XAOL6R-S7BFCM-6BDOLB-SZOXKGIY),CaseIndexDto(uuid=SFPALU-Z4QORK-ZYPOEK-2QK32DUE),CaseIndexDto(uuid=RBMPYH-VBOOBC-BSEJKD-7KL7CMQQ),CaseIndexDto(uuid=V26N2S-5XKDTB-BMTJBS-HS7S2LAU)]", "agent": [ { "type": { "coding": [ { "system": "https://www.hl7.org/fhir/valueset-participation-role-type.html", "code": "humanuser", "display": "human user" } ] }, "who": { "identifier": { "value": "UOSUJW-BRSDJL-ZGSXEW-3XCUCLHI" } }, "name": "NatUser" } ], "source": { "site": "sormas.lu", "type": [ { "system": "http://terminology.hl7.org/CodeSystem/security-source-type", "code": "4", "display": "Application Server" } ] }, "entity": [ { "what": { "reference": "public java.util.List de.symeda.sormas.backend.caze.CaseFacadeEjb.getIndexList(de.symeda.sormas.api.utils.criteria.BaseCriteria,java.lang.Integer,java.lang.Integer,java.util.List)" }, "detail": [ { "type": "param", "valueString": "CaseCriteria(birthdateDD=null,birthdateMM=null,birthdateYYYY=null,caseClassification=null,caseLike=null,caseOrigin=null,caseUuidsForMerge=null,community=null,creationDateFrom=null,creationDateTo=null,dateFilterOption=By Date,dateTypeCalss=class de.symeda.sormas.api.caze.NewCaseDateType,disease=null,diseaseVariant=null,district=null,eventLike=null,facilityType=null,facilityTypeGroup=null,followUpStatus=null,followUpUntilFrom=null,followUpUntilTo=null,followUpVisitsFrom=null,followUpVisitsInterval=null,followUpVisitsTo=null,healthFacility=null,includeCasesFromOtherJurisdictions=false,investigationStatus=null,jurisdictionType=null,mustBePortHealthCaseWithoutFacility=null,mustHaveCaseManagementData=null,mustHaveNoGeoCoordinates=null,newCaseDateFrom=null,newCaseDateTo=null,newCaseDateType=null,onlyCasesWithDontShareWithExternalSurvTool=null,onlyCasesWithEvents=false,onlyCasesWithReinfection=null,onlyContactsFromOtherInstances=null,onlyEntitiesChangedSinceLastSharedWithExternalSurvTool=null,onlyEntitiesNotSharedWithExternalSurvTool=null,onlyEntitiesSharedWithExternalSurvTool=null,onlyQuarantineHelpNeeded=null,onlyShowCasesWithFulfilledReferenceDefinition=null,outcome=null,person=null,personLike=null,pointOfEntry=null,presentCondition=null,quarantineTo=null,quarantineType=null,region=null,reinfectionStatus=null,relevanceStatus=Active,reportDateTo=null,reportingUserLike=null,reportingUserRole=null,sourceCaseInfoLike=null,surveillanceOfficer=null,symptomJournalStatus=null,vaccinationStatus=null,withExtendedQuarantine=null,withOwnership=true,withReducedQuarantine=null,withoutResponsibleOfficer=null)" }, { "type": "param", "valueString": "0" }, { "type": "param", "valueString": "100" }, { "type": "param", "valueString": "[]" } ] } ] }Load case data
Update a case
Archive a case
Delete a case
Data History
Use case: A user observes incorrect data in a few cases. To understand how exactly this came to be it should be possible to extract the change history of the cases, including what exactly changed, at what point in time and by which user the change was made.
Goals:
Provide the information when and by whom a change was made
Provide what was changed / what the data looked like before and after the change
SORMAS uses temporal tables to provide a history of all data changes. These automatically create a copy of the previous status in a history table each time a database entry is changed and provide it with a validity period. This also makes it possible to query the status of the data at any time in the past with simple SQL queries.