Last updated: 
1 day 18 hours ago
Blog Manager
One of Jisc’s activities is to monitor and, where possible, influence regulatory developments that affect us and our customer universities, colleges and schools as operators of large computer networks. Since Janet and its customer networks are classified by Ofcom as private networks, postings here are likely to concentrate on the regulation of those networks. Postings here are, to the best of our knowledge, accurate on the date they are made, but may well become out of date or unreliable at unpredictable times thereafter. Before taking action that may have legal consequences, you should talk to your own lawyers. NEW: To help navigate the many posts on the General Data Protection Regulation, I've classified them as most relevant to developing a GDPR compliance process, GDPR's effect on specific topics, or how the GDPR is being developed. Or you can just use my free GDPR project plan.

Group administrators:

Incident response, logfiles and the GDPR

Tuesday, May 1, 2018 - 09:15

The Article 29 Working Party has recently highlighted the importance of detecting and mitigating information security breaches. One of the key tools in doing this is logfiles: the European Court of Justice in Breyer v Germany recognised the role of web server logs, the Article 29 Working Party guidelines mention logs and network flow data. Common questions are "what logs do I need?" and "how long should I keep them?". But that’s actually the wrong place to start.

Instead, look at your processes for detecting and responding to breaches. If you don't have such processes, write them. Those processes will tell you which logs you need: for attacks from outside, logs from externally visible servers and network flows are likely to be the foundation; for internal attacks, and to deal with complaints, you need to be able to translate the local IP and email addresses in those logs and reports into the identity of the individuals responsible for the problematic activity. Jisc's technical guide to logfiles has more detail. Exercises are a good way to test processes and logging: start from a (theoretical) report or alert (NIST Appendix A is a good source of example scenarios), and apply your response process to confirm that it works. That may highlight that you are missing some logs, tools or processes; or that you are unnecessarily keeping logs that no process will ever use.

As to how long to keep logs: clearly you need to keep them long enough that you'll be able to investigate the incidents that are discovered. Getting better at discovering your own incidents should mean that happens more quickly. Monitoring how long it takes you to detect incidents and then to fix them can be a useful indicator of the effectiveness of your incident response, as well as a guide to appropriate retention times. However surveys still show that incident detection can take several months, particularly in sectors or organisations where the majority of incidents are detected by third parties. If you are one of those, improving your own detection is a better approach than keeping logs longer in the hope that someone else will eventually find your incidents for you.

Under the General Data Protection Regulation, the justification for retaining logs is that the benefits to users of detection are greater than the risks caused to them by retention. But the longer an intruder stays in your systems undetected, the less benefit there will from a detailed investigation. Once the intruder has taken all the data accessible from a system, backdoored it to give them a full and persistent control, and modified applications and logs to conceal these activities, the benefit of all those logfiles is pretty small.

Many years ago, the UK Data Protection Commissioner suggested keeping routine logs for three to six months. Logs for incidents being investigated can obviously be kept for longer if required. At the time, we hoped that improved detection and investigation techniques would gradually let us reduce that period but, sadly, it still seems about right for most organisations.