Cloudflare ACE Training

Prerequisite

Hello World! This is Jean's test site for her partner engineers.
Please make use of this page if you happen to be faster or slower than my training speed.

FIRST - YOU NEED A CLOUDFLARE ACCOUNT

1. Work with Cloudflare Partner Team to settle your Partner Demo Account. This should have been done before you attend the ACE training. It can take 1~5 days to settle your demo license, so please request it with some room.
2. Once your demo account is ready, you should have access and can login to Cloudflare dashboard
3. Once login, you should be able to see Partner Demo Account, with one Enterprise Domain Quota given.




SECOND - YOU NEED A FREENOM ACCOUNT
We will use one external service: Freenom, they have free domain offering which we will use for Cloudflare onboarding.

4. Go to freenom.com or dot.tk and create your account.
5. Get a free domain at “services - Register new domain”.



Proceed if you can see this at Services/My Domains





THIRD - YOU NEED A TERMINAL FRIENDLY ENV

6. Prepare your own terminal-friendly environment.
   Make sure a basic curl command like below works and gives you output.
   $ curl -svo /dev/null/ https://www.google.com/
• If you use MacOS or Linux, just open your terminal would do.
• If you use Windows, follow this instruction to get your subsystem ready.

Please make sure you are ready for ACE exercises.

Onboarding

Goal: Complete Cloudflare Onboarding.
Activate Cloudflare account on CNAME setup. Onboard two hosts: staging and production.

a. Use a free registrar to get your domain.

1. Go to freenom.com and create your account.
2. Get a free domain at “services - Register new domain”.
3. Go to Cloudflare web dashboard.
4. Access “Partners Demo (Bootcamp) Account”.
5. Click “Add a site” and register your new domain.
6. Select Enterprise Plan.
7. Finish the guide.

This is what you would be seeing after the steps.

b. Cloudflare account activation on CNAME setup.

1. Go to Cloudflare dashboard - Overview.
2. Find “Advanced Options -- Convert to CNAME DNS Setup” at the bottom of the screen.
3. Read the warning and click “Convert”.


4. Find TXT record for the zone validation, then go to Freenom dashboard.
5. At Freenom, access “My domains - Manage Domain - Manage Freenom DNS” and add TXT record you were given.


6. Confirm the TXT record propagation at https://dnschecker.org. Then confirm Cloudflare account validation.

Congratulations, account actication success!

c. Health Checks CF<->Origin.

1. Go to Cloudflare dashboard.
2. Access Traffic - Health Checks.
3. Configure health checks. Today’s origin: 35.234.81.115.

You should always proceed to next steps after confirming your health check is successful.

Like this.

d. Activate Cloudflare for staging host.

1. Go to Cloudflare dashboard - DNS.
2. At Cloudflare DNS, add a DNS record as following:
   Type: A
   Name: staging
   IPv4 address: 35.234.81.115
   Proxy status: on (orange-clouded)


3. Go to Freenom DNS control panel.
4. At Freenom DNS, add a DNS record as following:
   Type: CNAME
   Name: staging
   Value: staging.yourdomain.com.cdn.cloudflare.net
   TTL: 300


5. Confirm the new DNS record propagation at https://dnschecker.org.
6. Confirm if your site is accessible at browser and/or terminal.
   $ curl -svo /dev/null/ http://staging.yourdomain.com 2>&1 | grep 'HTTP'
   HTTP/2 200 OK

e. Activate Cloudflare for the production host: www.

Activate Cloudflare for the "www" host. You can refer to the step d1-d6 if you are not sure how to.

f. Confirm Cloudflare is on.

• Use your terminal to send below two requests, and compare the difference in HTTP response headers.
  curl -svo /dev/null/ http://35.234.81.115/
  curl -svo /dev/null/ http://staging.yourdomain.com/
  curl -svo /dev/null/ https://www.yourdomain.com/
• Did you find the difference in response headers? Think about why.
• Are you able to connect to HTTPS to both sites? Think of a reason about why you can or can’t.

Send curl to the origin.

Send curl to Cloudflare onboarded domain.

Fastest DNS in the World

Jean keeps saying Cloudflare DNS is fastest in the world and we must recommend Cloudflare DNS (and full setup) to the customer, let's verify why.

• Go to DNS Speed Test Tool
• Try with non-Cloudflare domain. e.g. your customer domain
• Try with cloudflare.com

Example: DNS speed test to freenom.com.

Example: same to cloudflare.com.

Would you like to test more?
You may want to add a record at Freenom DNS (or your choice of any authoritative DNS) then meaure its propagation time vs add a record at Cloudflare DNS (you need a full setup zone for this) then measure its propagation time. You can measure it at dnschecker.org.

HTTPS Everywhere!

Cloudflare Certificate is easy to issue and maintain.

Already-available SSL Without Action

At your browser, connect to https://www.yourdomain.cf/ and see the certificate information.

Confirm HTTPS is already available without your action. See the certificate information.

Learn how to verify which certificate is shown at the browser.

Try the following commonly used SSL setting.

• Automatically redirect http:// requests to https://
• Order an Advanced Certificate with your preferred custom hostnames, validity period, certificate authority.
• Upload your own certificate in the dashboard.

Open Your Terminal!

Let's have some fun with basic commands you can use for Cloudflare.

dig

Dig is command line tool similar to nslookup that is used to run DNS queries and check DNS records for a given domain/website.
$ dig yourdomain.cf

Try it with your domain.

curl

cURL is a command line tool used to transport data using the URL syntax.
$ curl -svo /dev/null/ https://www.yourdomain.cf/

Try it with your domain.

$ curl -svo /dev/null/ http://www.yourdomain.cf/ --connect-to ::35.234.81.115

Use cURL option to check the origin response directly.

MTR

MTR/Traceroute is Network based command line tools used to measure performance/latency on a particular path to a given host/destination.
$ sudo mtr www.yourdomain.cf

Try it with your domain.

We are good for the Day 1! Thank you for your hard work!

Day 2 Start
Custom Error Pages

Let's see why we need "Custom Error Pages" for enterprise customers.

1. At Cloudflare dashboard, go to TLS/SSL - Overview and select “Strict (SSL only origin pull)”.
2. At the browser, visit your website: https://www.yourdomain.cf/



Think about why you see this error.

3. At the browser, access http://www.ace-training.cf/error.html
4. At Cloudflare dashboard, visit Custom Pages - 500 Class Errors, and try publishing the page: http://www.ace-training.cf/error.html
5. Give Cloudflare some time to provision, then visit your website again: https://www.yourdomain.cf/ and confirm the difference.
6. Once you're finished with testing, please roll back the SSL setting to "full".

Why configuring custom error pages is a must? When something happens, what do you think the customer wants to show - their branded page vs Cloudflare branded error page?

Configure Alerts

Get alerted by email in the event of: Origin server failure, DDoS attack

1. At Cloudflare dashboard, go to Account Home - Notification.
2. Try setting ‘HTTP DDoS Attack Alert’, 'Layer 3/4 DDoS Attack Alert' and ‘Passive Origin Monitoring’ to your work email.
3. Once alert is set, you will get alerted on HTTP DDoS events, and origin health (even if no active health check) based on limited set of passive error codes. (e.g. 521)

Note:

• Alert only works after you have added the notifications;
• Layer 3/4 DDoS Attack Alerts is currently available for Magic Transit, Spectrum customers only;
• Customizing alerting threshold is not supported

Origin Correctly Secured?

1. Try access https://pentest-tools.com/network-vulnerability-scanning/tcp-port-scanner-online-nmap
2. Enter the test origin address 35.234.81.115 and run a light scan. Check the result.
3. If you have, check your own test origin server or a customer's origin server.
4. Do you think this origin is opening the web port to ANY? Were you able to confirm?
5. Even if the origin is set like this, if it’s behind Cloudflare, is it immune to DDoS and security threats?

What should you advise to the owner of this server?

With vs Without Our WAF

1. Access Cloudflare dashboard - Firewall - Managed Rules. Confirm WAF is OFF.
2. Access https://www.yourdomain.cf/file.php?cmd=echo(shell_exec(%22ls%20/etc/var%22))
   $ curl -svo /dev/null/ "https://www.yourdomain.cf/file.php?cmd=echo(shell_exec(%22ls%20/etc/var%22))"
3. Turn the WAF on.
4. Re-try the exploit.
5. (Optional) Try many times
   $ for i in {1..100}; do curl -svo /dev/null/ -H "exploit: true" "https://www.yourdomain.cf/file.php?cmd=echo(shell_exec(%22ls%20/etc/var%22))" 2>&1 | grep "< HTTP"; done;
6. Find the blocked logs at Cloudflare Firewall dashboard.

Check the detail of the event.

"I am Under Attack Mode"

1. At Cloudflare dashboard, adjust your test domain’s security level to 'I’m Under Attack Mode'.
2. At your browser, try access the domain and check the result and status codes.
3. At your terminal, try access the domain and check the result and status codes.
4. After the exercise, turn off the IUAM (change it to medium/high) so the next exercise won’t be disturbed.

Make sure you know what is this for.
Do you think the security level should be on if the customer has a public API endpoint?

Rate Limiting

Rate Limiting is a must for any clients who are worried about DDoS attack.

• At your terminal, run this command to send 200 requests to the site:
  for i in {1..200}; do curl -svo /dev/null/ -H "requestflood: true" "https://www.yourdomain.cf/" 2>&1 | grep "< HTTP"; done;
• Confirm your requests are served.
• At Cloudflare dashboard, create a Rate Limiting rule.


• Run the same command again.
  for i in {1..200}; do curl -svo /dev/null/ -H "requestflood: true" "https://www.yourdomain.cf/" 2>&1 | grep "< HTTP"; done;
• See what happens to your flooded requests.


• At Cloudflare dashboard, check the logs created just now.

Customize Security for Needs

1. Try set a rule to only allow one client IP ‘1.2.3.4’ when people access http(s)://yourtestdomain.cf/admin.html.
2. Try set a rule to give JS challenge to everyone when the request to http(s)://yourtestdomain.cf/* is NOT coming from your country.

Hint: You may want to check "Firewall Rules"

@_@

To Make the Magic Happen

1. Access https://webpagetest.org
2. Run a test to https://www.yourdomain.cf/ (choose a node you want; e.g. Singapore EC2)
3. Open another tab with same URL webpagetest.org, run another test to https://www.ace-training.cf/
   This is a same page proxying to the same origin. Use the same location with above (e.g. Singapore EC2)
4. Compare two results. Content is same. Is the result similar or different? Let’s think about why.

Think about why the difference between this and that

Challenge: Make your domain as fast as ace-training.cf too!

Customize CF Cache

Try creating a caching rule for all HTML pages under the domain.

• At the Edge:
• If status code == 200(OK), cache 1 minute
• If status code == 403(Forbidden) or 404(Not Found), cache 10 minutes
• At client browser: Recommend cache for 12 hours

Hint: Cache customization has to be done at "Page Rules"

Optimization Quick Summary

• ONBOARDING
• Set up health checks
• Set up necessary notification email
• Use custom error pages
• Consider Cloudflare dashboard 2FA or SSO
• Origin to allow-list Cloudflare IPs
• Review CF headers/cookies, restore the originating IP if needed.
• Enable CF Logpush (or Logpull)
• SECURITY
• Avoid HTTP, redirect/rewrite to HTTPS
• Hide origin IP/port
• Turn on WAF
• Use Security Level at Med or High, IUAM if needed.
• Set Rate Limiting Rules
• Get bots visibility and control them
• Deploy their security needs at Cloudflare Firewall Rules
• PERFORMANCE
• Cache as much as possible
• Maximise use of optimization features
• Use Brotli compression
• Use Argo Smart Routing
• Use newer TLS and better technologies
• Keep the customer’s staging subdomain so you can always use it for origin compatibility test vs. CF optimization feature

We are good for the Day 2! See you tomorrow!

DAY 3 START
TURN ON THE LOG

Cloudflare logs is enterprise only, and it has all necessary information you need for 99% of troubleshooting. While Cloudflare recommends Logpush over Logpull (remember this on customer project), we will be using Logpull for this lab because not everyone has access to cloud based storage.

• Follow this doc https://developers.cloudflare.com/logs/logpull-api/enabling-log-retention to enable Log retention. You can use Postman or Terminal.
  

  
  Confirm "flag": true, "success": true

Compare with cURL

Use cURL resolve override to compare the behaviour between response-coming-via-Cloudflare vs response-directly-from-origin.

1. curl -svo /dev/null/ "http://www.yourdomain.cf"
2. curl -svo /dev/null/ "http://www.yourdomain.cf" --resolve www.yourdomain.cf:80:35.234.81.115
3. Think about in what situation you can make use of these two commands.

Visit cdn-cgi Page

When a host is on Cloudflare there's a way to find metadata easily.

1. At Cloudflare dashboard, grey-cloud staging.yourdomain.cf
2. At the browser, visit http://www.yourdomain.cf/cdn-cgi/trace/
3. At another tab of the browser, visit http://staging.yourdomain.cf/cdn-cgi/trace/

HAR File to get empathy!

Generate HAR file to show the status code and a page you see. If the error is harder to replicate, it will help the Cloudflare support team to understand what you are seeing, or what your customer is seeing.

1. At your browser, open developers tool first, access http://www.yourdomain.cf/
2. Right-click the mouse and find "Save as HAR with Content". Save it in your local.
   
3. Open Google HAR Analyzer, open the HAR file you have just saved.

Retrieve Logs with RayID

RayID is Cloudflare's identifier of a request. Use RayID to retrieve detailed information of a request.

1. rayId: Find one RayID from your zone
2. zoneName: yourdomain.cf
3. Guidance and example: here
4. Hint: Start from a readymade query
5. Log fields and its meaning: https://developers.cloudflare.com/logs/log-fields

Easier and quicker with POSTMAN.

Give me all logs for the last 1h

Try to use timestamp to download all logs of a Cloudflare ENT zone within specific time range.

1. zoneName: yourdomain.cf
2. time_start: Use this format [ 2021-02-25T03:00:00Z ]
3. time_end: Use this format [ 2021-02-25T04:00:00Z ]
4. Guidance and example: here
5. Hint: Start from a readymade query
6. Log fields and its meaning: https://developers.cloudflare.com/logs/log-fields

Easier and quicker with POSTMAN.

Traceroute from Cloudflare Colo!

If the feature is enabled, you can use API to run traceroute from specific Cloudflare data center to the origin server.

1. Access https://api.cloudflare.com/#diagnostics-properties
2. Try running a diagnostics to: 35.234.81.115 from: SIN02.
3. Hint: Account ID can be found in dashboard overview or at the address line.

Easier and quicker with POSTMAN.

Play with cURL

Directly send request to origin, repeat cURL 50 times to confirm intermittent error, send specific headers, etc.

• for i in {1..50}; do curl -svo /dev/null/ "https://www.cloudflare.com?x=${i}" 2>&1 | grep -Ei "< HTTP|< Date|< CF-Cache-Status|< CF-RAY|< Server"; printf "\n\n"; done;
• curl -o /dev/null -D- -k -s -v -w '%{url_effective} CODE:%{http_code} \n -- DNS:%{time_namelookup} \n -- Connect:%{time_connect} \n -- TTFB:%{time_starttransfer} \n -- Time_Pretransfer:%{time_pretransfer} \n -- Time_Redirect:%{time_redirect} \n -- Time_Start_Transfer:%{time_starttransfer} \n -- TOTAL Time:%{time_total}\n -- Size_download:%{size_download} \n' "https://cloudflare.com"

Troubleshooting Quick Summary

1. Initial t/s investigation check points:
• Health checks
• CF-Origin response codes cross check (cURL)
• Confirm logs of the corresponding RayID
• Change logs (audit logs) of the problematic time
• Check cloudflarestatus.com
• Save cdn-cgi and HAR
2. Still can’t find root cause? SOS to the below contact
• Customer issue: [email protected]
• Partner issue: [email protected]
• IMPORTANT: Share the initial t/s result. It will save time!
• IMPORTANT2: Need to contact support from the right legitimate email.

Challenges!

This challenge is for the trainees who finished the day 1-3 exercises.
Make sure you are equipped with necessary knowledge by completing the challenge.

Spin up your own origin server (cloud, on-prem, doesn't matter) and configure the below using your domain used for the ACE exercises.

Onboarding Requirements

1. Make sure the following hosts are acceessible via Cloudflare:
   https://web.yourdomain.com/,
   https://api.yourdomain.com/,
   https://admin.yourdomain.com/, and
   ssh.yourdomain.com
2. Protect the web service at 80/443, and SSH service at 22.
3. Make sure nmap test says the origin doesn't have 22/80/443 opened.
4. Make sure an email alert will be sent when the origin is not reachable from Cloudflare.
5. Make sure web.yourdomain.com has following HA set up:
   All visitors will be routed to the primary origin. But once the primary origin is not reacheable, it will fallback to the backup origin(35.234.81.115)

Security Requirements

1. Any requests over HTTP needs to be redirected to HTTPS.
2. Make sure there's no path of HTTP traiffic flows unencrypted.
3. Make sure you use Cloudflare's WAF and IP Intelligence.
4. Make sure you block anyone who sends more than 600 requests in a minute. Blocking time is 5 minutes.
5. Site Rule: admin.yourdomain.com should NOT allow any requests without the following request header 'admin: true'
6. Site Rule: admin.yourdomain.com should NOT allow any external access other than my own IP
7. Site Rule: api.yourdomain.com should NOT use Cloudflare's WAF and IP intelligence.
8. Site Rule: Anyone who accesses web.yourdomain.com/country.html from anywhere but your country or residence, should firstly pass JS Challenge.

Performance Requirements

1. Check the current cache TTL and change it to 7 days.
2. Cache the front page for 1 minute.
3. Optimize the content as much as possible. Compress the images and codes.
4. What other settings do you need to enable to make the site faster? Find and enable them. (Hint: Refer to optimization training)

API and Troubleshooting

1. Set the WAF in "Detection-only" mode.
2. Set up Logpush for HTTP Request, and Spectrum Events.