How establish a raw connection over HTTP/HTTPS
Danilo Novais
Software Engineer @ BairesDev | BTL1 | .NetCore | C# | Vue.js | Python | Cyber Security | Ethical Hacker
When we talk about network connection or communication we cannot stop talking about protocols. Network protocols may initially appear confusing or even misunderstood.
A brief formal definition of what a network protocol would be:
"A network protocol is an established set of rules that determine how data is transmitted between different devices in the same network."
In other words, protocols are basically a set of rules that both parties agree in order to establish a communication.
A complete definition of a protocol should indicate in detail what the client (consumer) is expected to send to the server (provider) and how the server should respond to that request.
They are, for example, concerns that a protocol may have to guarantee the delivery and receipt of the message between the different systems.
Given that we have HTTP/ HTTPS protocols
Today we won't talk about how these protocols behave or their specifics, but how we can establish a connection with the server speaking their languages.
For this purpose we'll to use a linux based operating system, but all samples can be replicated in any other operating system. Initially we'll to use a tool called "netcat".
Netcat is a fantastic tool that we can use to establish a connection with any server in any port of that server, if of course that server supply a service in that port (that port is open).
领英推荐
$ nc <domain> <port>
$ nc www.example.com 80
GET / HTTP/1.1 <---------- request for path / using http protocol version 1.1
Host: www.example.com <--- host header used internally in the server for redirections
HTTP/1.1 200 OK <--------- response status code 200 indicating success
Age: 440797
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Sat, 08 May 2021 19:30:14 GMT
Etag: "3147526947+ident"
Expires: Sat, 15 May 2021 19:30:14 GMT
Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
Server: ECS (mic/9A9C)
Vary: Accept-Encoding
X-Cache: HIT
Content-Length: 1256
<!doctype html>
<html>
<head>
? ? <title>Example Domain</title>
...SNIP...
</head>
<body>
<div>
? ? <h1>Example Domain</h1>
? ? <p>This domain is for use in illustrative examples in documents. You may use this
? ? domain in literature without prior coordination or asking for permission.</p>
? ? <p><a >More information...</a></p>
</div>
</body>
</html>
So, we were able to establish a connection to "www.example.com" domain using the tool netcat with success.
But, if we try establish a connection to another domain say, "www.dhirubhai.net":
$ nc www.dhirubhai.net 80? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
GET / HTTP/1.1
Host: www.dhirubhai.net
HTTP/1.1 301 Moved Permanently <------- redirection status code
Location: https://www.dhirubhai.net/ <-- domain and protocol we have to use
X-Li-Pop: afd-prod-eda6
X-Li-Proto: http/1.1
X-LI-UUID: D2tI6X0vfRaAws6sDysAAA==
X-Cache: CONFIG_NOCACHE
X-MSEdge-Ref: Ref A: 2CD859BC21C04D34BCB5C601ADA9C7CD Ref B: SAO03EDGE0519 Ref C: 2021-05-08T19:54:18Z
Date: Sat, 08 May 2021 19:54:18 GMT
Content-Length: 0
The server replied to us with a redirection status code (301) and we must to use HTTPS instead of HTTP for the connection. The HTTPS protocol is HTTP over SSL protocol used for create a tunnel with the server and our message is encrypted in that communication. This means nobody can access this information beside us or the server.
But the tool netcat can't establish a connection over SSL for us. So, we'll to use another tool called "openssl". Openssl is a complete tool for deal with all aspects of SSL protocol. In this example we'll to use openssl to establish a tunnel with the server, so after that we can use HTTP to send information, establishing a complete HTTPS.
For establish a connection over https the default port used is 443.
$ openssl <command> <domain>:<port>?
$ openssl s_client www.dhirubhai.net:443?
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
verify return:1
depth=0 C = US, ST = California, L = Sunnyvale, O = LinkedIn Corporation, CN = www.dhirubhai.net
verify return:1
---
Certificate chain
?0 s:C = US, ST = California, L = Sunnyvale, O = LinkedIn Corporation, CN = www.dhirubhai.net
? ?i:C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
?1 s:C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
? ?i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIH6DCCBtCgAwIBAgIQB7TyNplLvd1iFgIrbv98MzANBgkqhkiG9w0BAQsFADBN
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMScwJQYDVQQDEx5E
aWdpQ2VydCBTSEEyIFNlY3VyZSBTZXJ2ZXIgQ0EwHhcNMjEwMjE3MDAwMDAwWhcN
MjEwODE2MjM1OTU5WjBwMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5p
YTESMBAGA1UEBxMJU3Vubnl2YWxlMR0wGwYDVQQKExRMaW5rZWRJbiBDb3Jwb3Jh
dGlvbjEZMBcGA1UEAxMQd3d3LmxpbmtlZGluLmNvbTCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAJxXbQKRgGC+o5EOLWkSNwZ5ZlUX5MKNABnP52WilnSC
9S3x/mxJkzAvWsoXbHYvT4CxSYPNPeMnLtgNsKOfb6c/CNmPuFvGcLGvweAtHUSF
mgyfhJ3qvllme1r3/cNeZcfW4WNa++J2rQrJdF1vpFrlCa4pyQ/kDeCxb7qMs3a1
Vq6RYoxC6C00BeAz39MqVg06thDQ7hr5U85Czfiqjr6l3mptFs0Xq/oUchwCtQSf
fLEJHYvIZbbWUU2eDWsqFm7oOnWpvG9NLwvHy6sleOMoex9mG1FsdY9+d8x1iAi9
29O8X4fk6RM/wq67fHK7JqoJY19KxE+nU5MwrLWkIVkCAwEAAaOCBJ8wggSbMB8G
A1UdIwQYMBaAFA+AYRyCMWHVLyjnjUY4tCzhxtniMB0GA1UdDgQWBBR0SiN/mkvY
PRo3APRFcNFcfvmdLTCCAcYGA1UdEQSCAb0wggG5ggxsaW5rZWRpbi5jb22CEHd3
dy5saW5rZWRpbi5jb22CFnJ1bTUucGVyZi5saW5rZWRpbi5jb22CFWV4cDQud3d3
LmxpbmtlZGluLmNvbYIVZXhwMy53d3cubGlua2VkaW4uY29tghVleHAyLnd3dy5s
aW5rZWRpbi5jb22CFWV4cDEud3d3LmxpbmtlZGluLmNvbYIWcnVtMi5wZXJmLmxp
bmtlZGluLmNvbYIWcnVtNC5wZXJmLmxpbmtlZGluLmNvbYIWcnVtNi5wZXJmLmxp
bmtlZGluLmNvbYIXcnVtMTcucGVyZi5saW5rZWRpbi5jb22CFnJ1bTgucGVyZi5s
aW5rZWRpbi5jb22CFnJ1bTkucGVyZi5saW5rZWRpbi5jb22CFWFmZC5wZXJmLmxp
bmtlZGluLmNvbYIXcnVtMTQucGVyZi5saW5rZWRpbi5jb22CF3J1bTE4LnBlcmYu
bGlua2VkaW4uY29tghdydW0xOS5wZXJmLmxpbmtlZGluLmNvbYIVZXhwNS53d3cu
bGlua2VkaW4uY29tghlyZWFsdGltZS53d3cubGlua2VkaW4uY29tMA4GA1UdDwEB
/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwgYgGA1UdHwSB
gDB+MD2gO6A5hjdodHRwOi8vY3JsMy5kaWdpY2VydC5jb20vRGlnaUNlcnRTSEEy
U2VjdXJlU2VydmVyQ0EuY3JsMD2gO6A5hjdodHRwOi8vY3JsNC5kaWdpY2VydC5j
b20vRGlnaUNlcnRTSEEyU2VjdXJlU2VydmVyQ0EuY3JsMD4GA1UdIAQ3MDUwMwYG
Z4EMAQICMCkwJwYIKwYBBQUHAgEWG2h0dHA6Ly93d3cuZGlnaWNlcnQuY29tL0NQ
UzB+BggrBgEFBQcBAQRyMHAwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2lj
ZXJ0LmNvbTBIBggrBgEFBQcwAoY8aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29t
L0RpZ2lDZXJ0U0hBMlNlY3VyZVNlcnZlckNBLTIuY3J0MAwGA1UdEwEB/wQCMAAw
ggEFBgorBgEEAdZ5AgQCBIH2BIHzAPEAdwD2XJQv0XcwIhRUGAgwlFaO400TGTO/
3wwvIAvMTvFk4wAAAXexgaGBAAAEAwBIMEYCIQD8E+f1VNmw1o7gEW4ijwTDpO3n
Rvk4UVfhSA4DrRuKVAIhAJF1lGRlnQ9ThCCFo77oym5kCoCMwSB0XK74TsCm6CHW
AHYAXNxDkv7mq0VEsV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF3sYGh1gAABAMA
RzBFAiEA5+vMt9uR8oCa3MssyEmHfiQnONLVnooNk7owgiiDpncCIAo2WZ6AD7Zz
8T5IXcinp9GM5ET0xVpLT52D/54KWNNEMA0GCSqGSIb3DQEBCwUAA4IBAQA+gvKQ
KeLI6LtpsOKXUS+bfW734DDNwYswer5x1YBqbp/sLNGxcfOsgwaPTZVWafDI3bFB
8+2Bb8fqbzFcWmw/LwxTpeGmIIUYAMM3y6yMaaF/3m54yUaH6OUVv1fHVxJAAsc7
ThbYH42stwdwLUzK4rMmHtjkW1oZajRtTp4i4Q5pa3pc6DuGtHqRKBXDpYMzsUUH
Nsro6y18A5NJIPCOEe2Un7aj4HYGhKVL/ld5R7BAtfSEzrVdnL9teF7jsgiT/ook
UQ5cSBuKEaiwFHxGP7JnpiNtUoK15Bidj7H1wMzlFtoCizPSOzwhEUqgtwr4X7cH
u+Wx7X+UAJBAD4Yo
-----END CERTIFICATE-----
subject=C = US, ST = California, L = Sunnyvale, O = LinkedIn Corporation, CN = www.dhirubhai.net
issuer=C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 4090 bytes and written 466 bytes
Verification: OK <------------ connection established
---
How we can see our connection was established with domain "www.dhirubhai.net" over SSL, so, we can type our request to the server now.
...SNIP... <------------- certificate and SSL handshake information (above)
GET / HTTP/1.1
Host: www.dhirubhai.net
HTTP/1.1 200 OK <------- response status code 200 indicating success
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Length: 89382
Content-Type: text/html; charset=utf-8
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: JSESSIONID=ajax:4633979922463481702; SameSite=None; Path=/; Domain=.www.dhirubhai.net; Secure
Set-Cookie: lang=v=2&lang=en-us; SameSite=None; Path=/; Domain=linkedin.com; Secure
Set-Cookie: bcookie="v=2&55335426-7cc2-4ae3-80ca-7e6308c56eec"; domain=.linkedin.com; Path=/; Secure; Expires=Tue, 09-May-2023 08:12:51 GMT; SameSite=None
...SNIP...
X-Frame-Options: sameorigin
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000
Expect-CT: max-age=86400, report-uri="https://www.dhirubhai.net/platform-telemetry/ct"
X-Li-Fabric: prod-lor1
X-Li-Pop: afd-prod-lor1
X-Li-Proto: http/1.1
X-LI-UUID: IEUOzboxfRawOiFORCsAAA==
X-Cache: CONFIG_NOCACHE
X-MSEdge-Ref: Ref A: 49ED189FE40749E8A3EF89AD3FD528BF Ref B: SAO03EDGE0619 Ref C: 2021-05-08T20:35:19Z
Date: Sat, 08 May 2021 20:35:18 GMT
<!DOCTYPE html>
? ? <html lang="en">
? ? ? <head>
...SNIP...
So, finally we were able to establish a connection with the server over HTTPS.
Thank you for reading and we see each other in the next article!