We Can (Not) Secure
Introduction to SecDevOps!
Best practices for securing large scale Devops environments. Introducing security earlier in the CICD pipeline.
Hi, I'm Shlee
and I Take your Security Seriously.
@CyberShambles
Devops Engineer for Versent. Speaking for myself.
This talk is dedicated to
Segment-your-network Woman
and Patch-your-shit Man!
Infracode loves Security
This talk is heavily focused on the cloudy AWS heavy environments.
Similar patterns can be used in similar "Cloud Orchestrator" Tools
HarshiCorp Terraform, OpenStack Heat, Atlas and others.
Everybody has their own view on what DevOps actually means to them.
My loose interpretation of "DevOps" is the idea of
source control, testing, deployment and code reuse ;)
Good Buzzwords
- Defense in depth
- Isolation/Seperation/segmentation
- Principle of least privilege (ACL)
- Cattle, not pets
- CloudFormation: IaaS Templates
- Elastic Compute Cloud "EC2": Xen based Virtual Machines
- Elastic Block Store (EBS): Block level storage
- Simple Storage Service (S3): Networked Storage
- Route 53: DNS (Public and Private Zones)
- Virtual Private Cloud (VPC): isolated networks
- Elastic Load Balancing (ELB): TCP Frontend
- Relational Database Service (RDS): MySQL
- CloudWatch: Monitoring log streams
- CloudTrail: Monitoring AWS API
- Identity and Access Management (IAM): Least privledge
- Simple Queue Service (SQS): One way messaging
Development
Operations
Security
Deployment
Infracode
Test early. Test Often. Fail quick.
Developer Duck
Commits to Git
Jenkins Triggers
Builds Application
Jenkins Triggers
static code analysis
Checkmarx pvs-studio coverity
3
1
2
Unit Tests
Developer Duck
Commits to Git
Jenkins Triggers
Builds Application
Cloudformation
Builds Infra
App Deployed
on SOE
Jenkins Triggers
Builds staging
3
1
2
New environment
on trusted SOE
on 'production' infracode.
Developer Duck
Commits to Git
Jenkins Triggers
Builds Application
Cloudformation
Builds Infra
App Deployed
on SOE
Jenkins Triggers
Builds staging
Arachni Vega
Zap Selenium
Jenkins Triggers
Security scans
3
4
1
2
DevSecOps
Scope, coverage and automation
Reporting/Alerts for automated scanning tools?
Static Application Security Testing (SAST)
Static Code
Dynamic Application Security Testing (DAST)
Security Scanners
Interactive Application Security Testing (IAST)
ModSecurity/WAF
Weirder shit
Fuzzing
DevSecOps
Scope, coverage and automation
Reporting/Alerts for automated scanning tools?
External Network Scans
nmap
Internal Network Scans
Nessus
Third party tools
Qualys SSLtest
DevSecOps
Scope, coverage and automation
Reporting/Alerts for automated scanning tools?
Reduplication? Delta reports?
Continuous reporting? Alerting?
No idea about that stuff.. ELK maybe?
Talk to me if you have any experience with this stuff.
1) Source Control
Maintain a solid git repository for as much as possible: Application Source/infracode/Jenkins/enviromental data.
For the following examples - Assume the following:
- SOE-RHEL72 Repo
- Shared Infracode Repo (Cloudformation & Jenkins)
- Environment Non-Production Repo
- Environment Production Repo
- Microservice (Java) Repo
Each of these Repos are accessible from inside the VPN
to people in the correct LDAP security groups.
+ Read-only access is given as required.
2) UPSERTING
ALL GLORY TO (INSERT OR UPDATE)!
3) One SOE to build them all
The "Hardened" SOE is the sidekick of "patch your shit man"
All of my AWS environment servers are based on a single AMI.
1) Harden single RHEL 7.2 gold release and
utilise in Dev,Test,Pre-Prod and Production.
2) Build AWS launch configuration userdata scripts
to maintain your freerange bespoke instances, built on SOE.
AWS RHEL Packer Serverspec SOE
- Bastion
- Frontend
- Backend
- Build
Tools: Packer by Hashicorp
Tweaking /etc/sysctl.cfg, confirming timezone, installing bastion keys (revoking older keys), run openscap-scanner
Tools: ServerSpec
Test suites meant to be run against machines
4) AWS Account isolation
"Dev put AWS keys on Github. Then BAD THINGS happened" - Darren Pauli
Seperate the business functions
into different accounts inside AWS.
- CloudOPS-Build for AMI/SOE
- CloudOPS for Management (Jenkins)
- Digital Prod/NonProd for Deployment
Seperate roles in each account to protect against stolen sessions!
Restrict functionality as required.
@dagrz loves popping AWS
5) VPC/Network Isolation
Cloudformation allows for the creation of new VPC and Subnets inside these VPCs easily.
Public subnets can be accessible from internet facing addresses
Private subnets can be accessible from public devices
Restricted can ONLY be accessed by private services with the correct security groups!
Preprod Production
6) Immutable infrastructure
Consider: Build new instances for each new version, Respond rapidly to security issues and reduce the risk of cowboys bugs.
Plus, existing ELBs are prewarmed from existing environment - Rolling upgrades work for webapps/api/microservices.
7) Secrets and AWS KMS Roles
KEEP YOUR PASSWORDS OUT OF THE SOURCE CONTROL FOR THE LOVE OF CHRIST! YOU GOD DAMN SAVAGES!
Tools: Credstash/Unicreds
KMS and Credstore dynamodb
Limit write access to passwords
KMS and Credstore dynamodb
Limit read access to passwords
8) Jenkins: Low Hanging Fruit
I love Jenkins, but it's a
perfect place to get persistence.
Using Jenkins Job Builder, I've moved on Jenkins code into the infracode repository.
Jenkins is read only and all modifications must be completed during the GIT repo.
Pull Requests preferred! :D
9) Untrusted Data and Docker
ImageTragick is an easy example of untrusted code running on production environments leading to RCE.
NEVER TRUST USER DATA!
10) AWS Cloudwatch
PUT IT ALL IN THE LOGS!
Graph EVERYTHING
https://codeascraft.com/2011/02/15/measure-anything-measure-everything/
11) Supermassive Black Holes
Can't afford an Threatbutt Advanced Enterprise Platform licence?
After all of the low hanging fruit has been found, it's time to give the professionals a call. You're going to have missed something!
Upright and proud
Thank you!
Questions?
SecDevOps Updated
By Shlee Paradiso
