You just need to set user agent header for it to work:
URLConnection connection = new URL("https://www.google.com/search?q=" + query).openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.connect();
BufferedReader r = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName("UTF-8")));
StringBuilder sb = new StringBuilder();
String line;
while ((line = r.readLine()) != null) {
sb.append(line);
}
System.out.println(sb.toString());
The SSL was transparently handled for you as could be seen from your exception stacktrace.
Getting the result amount is not really this simple though, after this you have to fake that you’re a browser by fetching the cookie and parsing the redirect token link.
String cookie = connection.getHeaderField( "Set-Cookie").split(";")[0];
Pattern pattern = Pattern.compile("content=\\\"0;url=(.*?)\\\"");
Matcher m = pattern.matcher(response);
if( m.find() ) {
String url = m.group(1);
connection = new URL(url).openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.setRequestProperty("Cookie", cookie );
connection.connect();
r = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName("UTF-8")));
sb = new StringBuilder();
while ((line = r.readLine()) != null) {
sb.append(line);
}
response = sb.toString();
pattern = Pattern.compile("<div id=\"resultStats\">About ([0-9,]+) results</div>");
m = pattern.matcher(response);
if( m.find() ) {
long amount = Long.parseLong(m.group(1).replaceAll(",", ""));
return amount;
}
}
Running the full code I get 2930000000L as a result.
Sometimes when trying to consume a web service from a java client, you get a 403 forbidden HTTP response code, even though the service is accessible normally from web browsers.
The HTTP 403 forbidden error doesn’t necessarily occur due to missing authentication attributes, some web services would only authorize web browsers or some specific clients to access them, while they deny any requests coming from third-party clients.
# Problem
I am getting HTTP 403 error when retrieving some data from a public web service, while i am able to retrieve them successfully when running the same URL link in web browsers.
This is indeed a very common problem which most developers face when consuming a web service.
# Resolution
This problem is normally resolved by imitating the web browser request so that the web service deals with the java client as if it was a web browser.
The first thing to do is to run the service URL on a web browser and use “live http headers” plugin (or any equivalent) to log the passed header attributes from the browser. Below is a typical request logged from a browser:
GET /feeds HTTP/1.1 Host: publicservice.com:443 Accept: */* Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9 Cookie: OGP=-4061129:; SID=FAYIU7tO.... Referer: https://clients5.google.com/pagead/drt/dn/ User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 X-Client-Data: CIa2yQEIpLbJAQjBtskBCKmdygEIqKPKARiSo8oB
As noticed, the “User-Agent” header specifies the name and the type of the client which is trying to access the service, so in order to imitate the web browser we need to add this header to our request. Following is how to add it using HttpUrlConnection:
String url = "https://publicservice.com/feeds";
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("Content-Type", "application/json");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36");
Hussein is a senior software engineer with 5 years of experience in software design, development and integration. He is the author and founder of Programmer Gate (www.programmergate.com) blog.
Back to top button
The HTTP 403 error in Java usually occurs when the web server forbids access to your web browser, hence the forbidden error. As a result, the HTTP status code confirms the flaws and inconsistencies and stops other processes, although most elements and functions indicate no obstacles in the main document.
In addition, an identical HTTP/1.1 403 forbidden message compromises your programming experience when the server owners set up the permissions incorrectly, forcing your system to show the 403 forbidden Spring Boot code exception. Hence, we suggest reading this ultimate guide because it teaches how to resolve 403 forbidden error in rest API Java with simple debugging methods and solutions that apply to all operating systems.
Contents
- What Causes the 403 Forbidden With Java but Not Web Browser?
- – The Server Forbids Access to Your Web Browser
- – The Server Owners Set up Invalid Permissions
- How to Fix the HTTP 403 Code Exception in Java? 3 Great Fixes
- – Adding a Request for the URL Connection Object
- – Passing the Headers With Attributes From the Browser
- Conclusion
What Causes the 403 Forbidden With Java but Not Web Browser?
The Java IO filenotfoundexception response 403 forbidden for URL code exception, is caused by denied access on your web browser. In addition, we confirmed an identical obstacle when the server owners set up the permissions for the web page incorrectly, preventing users from accessing the page.
The authorisation error 403 is typical with complex applications and programs, although we experienced it with simple projects. For example, the operating system displays a flawed message when it denies access to a specific server caused by the need for permissions and licenses.
Consequently, the HTTPURLconnection 403 error messages almost always happen due to issues with the current repository, although your syntax could have some inconsistencies and mistakes. Hence, we will teach you how to throw 403 forbidden exception in Java with common elements developers use when providing permissions and page’s structure.
On the contrary, we experienced a similar HTTP invalid status code when the server owners set up the permissions incorrectly. As a result, you cannot access the page or input values regardless of the syntax, which can be challenging to overcome, especially with secure paths and locations.
We suggest isolating these instances and troubleshooting the elements for more information because they provide critical information before implementing the debugging techniques and solutions. Nevertheless, although your document should raise slightly different visual outputs and warnings, the bug’s root and cause remain identical across all platforms and operating systems.
– The Server Forbids Access to Your Web Browser
This article confirmed the code exception is unavoidable when the server forbids access to your web browser. This inconsistency forces the program to show complex traceback calls that pinpoint the invalid code snippets, which is beneficial before implementing the debugging approaches.
Hence, this section exemplifies the main document with several imports and public classes that provide the project’s purpose and implementation. Finally, we will show you the debug logs that confirm the flaws.
You can learn more about the main document below:
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
public class DataGetter {
public static voids main (Strings[] args) throws IOException {
getResultAmount (“test”);
}
private static int getResultAmount (String query) throws IOException {
BufferedReader r = new BufferedReader (new InputStreamReader (new URL (“https://www.google.com/?q = ” + query) .openConnection()
.getInputStream()));
String line;
String src = “”;
while ((line = r.readLine()) ! = null) {
src + = line;
}
System.out.println (src);
return 1;
}
}
Although this syntax indicates no mistakes, it fails to complete the intended purpose and displays the 403 Java mistake. Still, the traceback calls should help you find the cause, as the following example shows:
at sun.net.http.HttpURLConnection.getInputStream (Unknown Source)
at sun.net.https.HttpsURLConnectionImpl.getInputStream (Unknown Source)
at DataGetter.getResultAmount (DataGetter.java: 15)
at sun.net.www.protocol.getInputStream (Unknown Source)
at sun.HttpsURLConnectionImpl.getInputStream (Unknown Source)
at DataGetter.getResult (DataGetter.java: 54)
at DataGetter.main (DataGetter.java: 185)
This code snippet indicates a few incorrect code lines, but your project could show a more complex visual output. However, you can remain subtle because the bug’s root remains unchanged for all operating systems and projects. In addition, the solutions work for all instances, but we will wait to discuss the potential changes.
– The Server Owners Set up Invalid Permissions
Another common example that ruins your project and halts further alterations happens when the server owners set up incorrect permissions. This example displays an identical error log that kills your program. Therefore, we will show you the syntax that does not work due to invalid permissions, although all inputs are functional.
The following example lists the dependencies and elements:
SyndFeed feed = null;
InputStream is = null;
try {
URLConnection openConnection = new URL(url).openConnection();
is = new URL (url) .openConnection() .getInputStream();
if (“gzip”.equals (openConnection.getContentEncoding())){
is = new GZIPInputStream (is);
}
InputSource source = new InputSource (is);
SyndFeedInput input = new SyndFeedInput();
feed = input.build (source);
} catch (Exception e) {
LOG.error (“Exception occured when building the feed object out of the url”, e);
} finally {
If ( is ! = null) is.close();
}
return feed;
}
Finally, it would help if you learned more about the traceback calls before implementing the solutions, as shown in the following example:
at sun.net.www.http.HttpURLConnection.getInputStream (HttpURLConnection.java: 17)
at org.podcastpedia.admin.service.UtilsImpl.getSyndFeedForUrl (UtilsImpl.java: 529)
at org.UpdateServiceImpl.getSyndFeedForUpdate (UpdateServiceImpl.java: 472)
at org.podcastpedia.admin.service.impl (UpdateServiceImpl.java: 389)
at org.podcastpedia.admin.service.implupdatePodcastById (UpdateServiceImpl.java: 221)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java: 39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java: 25)
at java.lang.reflect.Method.invoke (Method.java: 597)
We suggest comparing these inputs to your document to find the code lines that provoke your system. Nevertheless, nothing should compromise the solutions, especially after changing several values and inputs.
How to Fix the HTTP 403 Code Exception in Java? 3 Great Fixes
You can fix the HTTP 403 code exception in Java by setting a user agent header that handles the traceback calls. Other excellent debugging solutions suggest adding a request for the URL connection object or passing the headers with specific attributes from the browser.
We established several full-proof solutions that overcome the mistake and repair your program. Therefore, this section demonstrates the first debugging approach that requires the user agent header to handle the exceptions. Nevertheless, we worked on the alternative methods in the subsequent chapters.
You can discover the repaired syntax in the following example:
Pattern pattern = Pattern.compile (“content = \\\”0; url = (.*?) \\\””);
Matcher m = pattern.matcher (response);
If ( m.find() ) {
String url = m.group (1);
connection = new URL (url) .openConnection();
connection.setRequestProperty (“User-Agent”, “Mozilla/ 5.0 (Windows NT 6.1; WOW64) AppleWebKit/ 537.11 (KHTML, like Gecko) Chrome/ 23.0.95 Safari/ 537.11”);
connection.setRequestProperty (“Cookie”, cookie );
connection.connect();
r = new BufferReader (new InputStreamReaders (connection.getInputStream(), Charset.forName (“UTF-8″)));
sb = new StringBuilder();
while ((line = r.readLine()) != null) {
sb.append (line);
}
StringBuilder sb = new StringBuilder();
String line;
while ((line = r.readLine()) ! = null) {
sb.append (line);
}
System.out.println (sb.toString());
response = sb.toString();
pattern = Pattern.compile (” <div id = \”resultStats\” > About ([0-9, ] +) results </div>”);
m = pattern.matcher (response);
if ( m.find() ) {
long amount = Long.parseLong (m.group (1) .replaceAll (“, “, “”));
return amount;
}
}
This code snippet completes the initial debugging technique that overcomes the HTTP Java code exception without affecting other elements.
– Adding a Request for the URL Connection Object
This article’s second debugging section teaches you how to add a request for the URL connection object to clarify the code’s purpose. Regardless of the goal, you will remove the HTTP Java obstacle for all operating systems and versions.
Therefore, we will focus on the example from this article’s second broken chapter and repair its function with another request for the URL connection object. Again, we suggest pasting the element in your file, although several changes are required.
The following code snippet shows the corrected values:
SyndFeed feed = null;
InputStream is = null;
try {
URLConnection openConnection = new URL (url) .openConnection();
openConnection.addRequestProperty (“User-Agent”, “Mozilla/ 5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/ 2011 Firefox/ 25.0”);
is = openConnection.getInputStream();
if (“gzip”.equals s(openConnection.getContentEncoding())){
is = new GZIPInputStream (is);
}
InputSource source = new InputSource (is);
SyndFeedInput input = new SyndFeedInput();
feed = input.build (source);
} catch (ParsingFeedException e){
LOG.error (“************* ParsingFeedException *************\n “, e);
throw e;
} catch (MalformedByteSequenceException e) {
LOG.error (“************* MalformedByteSequenceException *************\n “, e);
throw e;
} catch (IOException e) {
LOG.error (“************* IOException *************\n “, e);
throw e;
} finally {
If ( is != null) is.close();
}
return feed;
}
The syntax appears merely identical, but the modifications are sufficient to repair the program and reenable all failed processes. We also demonstrated the script and confirmed no HTTP Java error logs should appear.
This article’s ultimate solution suggests passing the headers with specific attributes from the browser and clearing the paths. Still, you must replicate the immediate solution and provide the header plugin to allow the changes to occur before passing the headers.
Finally, you can request the changes, and the system will indicate the visual output from the browser. We will exemplify this output to access the service and imitate the solution.
We demonstrated the visual output in the following code snippet:
Host: publicservice.com: 443
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US, en; q = 0.9
Cookie: OGP = -4061129:; SID = FAYIU7tO….
Referer: https://clients5.google.com/ pagead/ drt/ dn/
User-Agent: Mozilla/ 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/ 537.36 (KHTML, like Gecko) Chrome/ 65.0.3325.181 Safari/ 537.36
X-Client-Data: CIa2yQEIpLbJ
String url = “https://publicservice.com/ feeds”;
URL obj = new URL (url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
con.setRequestMethod (“GET”);
con.setRequestProperty (“Content-Type”, “application/ json”);
con.setRequestProperty (“User-Agent”, “Mozilla/ 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/ 537.36 (KHTML, like Gecko) Chrome/ 65.0.3325.181 Safari/ 537.36”);
You have noted that this snippet resembles the values in the former sections, but some differences exist. For example, the header has unique attributes helping your program remove the flawed message. Still, you must replicate this debugging approach to all incorrect elements.
Conclusion
The HTTP 403 error log in Java usually occurs when the server forbids access to your web browser. Although many debugging methods exist, let us summarize this guide’s vital points before fixing your app:
- An identical code exception happens when the server owners set up the permissions incorrectly
- The debug logs and information are vital when troubleshooting the broken values
- You can fix the HTTP exception in Java by setting a user agent header that handles the debug logs
- Another excellent debugging solution suggests adding a request for the URL connection object
Many articles that teach several solutions for the HTTP 403 Java error logs exist, but this one is different. It replicates the mistake, demonstrates the fixes, and puts them into practice to help you overcome the obstacle.
- Author
- Recent Posts
Your Go-To Resource for Learn & Build: CSS,JavaScript,HTML,PHP,C++ and MYSQL. Meet The Team
The challenge – java.io.IOException: Server returned HTTP response code: 403 for URL
Remember my post Reading/Parsing RSS and Atom feeds in Java with Rome, remember how I build SyndFeed objects out of a feed’s URL:
public SyndFeed getSyndFeedForUrl(String url) throws MalformedURLException, IOException, IllegalArgumentException, FeedException {
SyndFeed feed = null;
InputStream is = null;
try {
URLConnection openConnection = new URL(url).openConnection();
is = new URL(url).openConnection().getInputStream();
if("gzip".equals(openConnection.getContentEncoding())){
is = new GZIPInputStream(is);
}
InputSource source = new InputSource(is);
SyndFeedInput input = new SyndFeedInput();
feed = input.build(source);
} catch (Exception e){
LOG.error("Exception occured when building the feed object out of the url", e);
} finally {
if( is != null) is.close();
}
return feed;
}
It works pretty well, but one bug managed to mangle through –
java.io.IOException: Server returned HTTP response code: 403 for URL https://happysomeone.com/feed/podcast:
java.io.IOException: Server returned HTTP response code: 403 for URL: https://happysomeone.com/feed/podcast at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1459) at org.podcastpedia.admin.service.utils.impl.UtilsImpl.getSyndFeedForUrl(UtilsImpl.java:529) at org.podcastpedia.admin.service.impl.UpdateServiceImpl.getSyndFeedForUpdate(UpdateServiceImpl.java:472) at org.podcastpedia.admin.service.impl.UpdateServiceImpl.getNewEpisodes(UpdateServiceImpl.java:389) at org.podcastpedia.admin.service.impl.UpdateServiceImpl.updatePodcastById(UpdateServiceImpl.java:221) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198) at sun.proxy.$Proxy23.updatePodcastById(Unknown Source) at org.podcastpedia.admin.controllers.UpdateController.updatePodcastsByIds(UpdateController.java:164) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:838) at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662)
This tells us that the web server may return a 403 Forbidden HTTP status code in response to the request I made and indicates that the server can be reached and understood the request, but refuses to take any further action. Status code 403 responses are the result of the web server being configured to deny access, for some reason, to the requested resource by the client.
The solution
At the same, if I access the feed url via the web browser, I don’t get any error messages and the feed data is displayed properly. Hmm, so what could the problem be? Let’s see what the GET request looks like via web browser:
GET /feed/podcast/ HTTP/1.1 Host: happysomeone.com User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-gb,en;q=0.5 Accept-Encoding: gzip, deflate Cookie: _ga=GA1.2.989262483.1379652403; __qca=P0-1317688601-1379652402575 Connection: keep-alive
Note: I am using the firefox plugin Live HTTP Headers to see this – very useful tool by debugging.
This led me to ask myself how can I make my request from Java more “browsery-like”. The first thing that comes to mind is trying to add an user agent to the request just like the one in line 3 from the web browser request. In Java this is possible by adding a request property to the URL connection object. The modified code looks like this:
public SyndFeed getSyndFeedForUrl(String url) throws MalformedURLException, IOException, IllegalArgumentException, FeedException {
SyndFeed feed = null;
InputStream is = null;
try {
URLConnection openConnection = new URL(url).openConnection();
openConnection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");
is = openConnection.getInputStream();
if("gzip".equals(openConnection.getContentEncoding())){
is = new GZIPInputStream(is);
}
InputSource source = new InputSource(is);
SyndFeedInput input = new SyndFeedInput();
feed = input.build(source);
} catch (ParsingFeedException e){
LOG.error("************* ParsingFeedException *************\n ", e);
throw e;
} catch (MalformedByteSequenceException e) {
LOG.error("************* MalformedByteSequenceException *************\n ", e);
throw e;
} catch (IOException e) {
LOG.error("************* IOException *************\n ", e);
throw e;
} finally {
if( is != null) is.close();
}
return feed;
}
Notice the method addRequestProperty in line 9. This adds a general request property specified by a key-value pair. This method will not overwrite existing values associated with the same key. In our case I set the user agent that Firefox uses.
Et voilà, this solved the problem. I hope you could learn something from this as I did.
Source code for this post is available on Github — podcastpedia.org is an open source project.
Creator of Podcastpedia.org and Codepedia.org, computer science engineer, husband, father, curious and passionate about science, computers, software, education, economics, social equity, philosophy — but these are just outside labels and not that important, deep inside we are all just consciousness, right?
Unexpected HTTP 403 error in Java Examples
In this lesson, we’ll use programming to attempt to solve the Unexpected HTTP 403 error in Java Examples puzzle. This is demonstrated by the code below.
/**
* This method will take the supplied URL String regardless of the protocol (http or https)
* specified at the beginning of the string, and will return whether or not it is an actual
* "http" (no SSL) or "https" (is SSL) protocol. A connection to the URL is attempted first
* with the http protocol and if successful (by way of data acquisition) will then return
* that protocol. If not however, then the https protocol is attempted and if successful then
* that protocol is returned. If neither protocols were successful then Null is returned.<br><br>
*
* Returns null if the supplied URL String is invalid, a protocol does not
* exist, or a valid connection to the URL can not be established.<br><br>
*
* @param webLink (String) The full link path.<br>
*
* @return (String) Either "http" for Non SLL link, "https" for a SSL link.
* Null is returned if the supplied URL String is invalid, a protocol does
* not exist, or a valid connection to the URL can not be established.
*/
public static String isHttpOrHttps(String webLink) {
URL url;
try {
url = new URL(webLink);
} catch (MalformedURLException ex) { return null; }
String protocol = url.getProtocol();
if (protocol.equals("")) { return null; }
URLConnection yc;
try {
yc = url.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
in.close();
return "http";
} catch (IOException e) {
// Do nothing....check for https instead.
}
try {
yc = new URL(webLink).openConnection();
//send request for page data...
yc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
yc.connect();
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
in.close();
return "https";
} catch (IOException e) {
// Do Nothing....allow for Null to be returned.
}
return null;
}
The same problem Unexpected HTTP 403 error in Java Examples can be solved in another approach that is explained below with code examples.
// Note that the http protocol is supplied within the url string:
String protocol = isHttpOrHttps("http://checkdnd.com/api/check_dnd_no_api.php?mobiles=9999999999");
System.out.println(protocol);
/**
* Returns a List ArrayList containing the page source for the supplied web
* page link.<br><br>
*
* @param link (String) The URL address of the web page to process.<br>
*
* @return (List ArrayList) A List ArrayList containing the page source for
* the supplied web page link.
*/
public static List<String> getWebPageSource(String link) {
if (link.equals("")) { return null; }
try {
URL url = new URL(link);
URLConnection yc = null;
//If url is a SSL Endpoint (using a Secure Socket Layer such as https)...
if (link.startsWith("https:")) {
yc = new URL(link).openConnection();
//send request for page data...
yc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
yc.connect();
}
//and if not a SLL Endpoint (just http)...
else { yc = url.openConnection(); }
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
String inputLine;
List<String> sourceText = new ArrayList<>();
while ((inputLine = in.readLine()) != null) {
sourceText.add(inputLine);
}
in.close();
return sourceText;
}
catch (MalformedURLException ex) {
// Do whatever you want with exception.
ex.printStackTrace();
}
catch (IOException ex) {
// Do whatever you want with exception.
ex.printStackTrace();
}
return null;
}
String netLink = "http://checkdnd.com/api/check_dnd_no_api.php?mobiles=9999999999";
String protocol = isHttpOrHttps(netLink);
String netLinkProtocol = netLink.substring(0, netLink.indexOf(":"));
if (!netLinkProtocol.equals(protocol)) {
netLink = protocol + netLink.substring(netLink.indexOf(":"));
}
List<String> list = getWebPageSource(netLink);
for (int i = 0; i < list.size(); i++) {
System.out.println(list.get(i));
}
{"msg":"success","msg_text":{"9999999999":"N"}}
Many examples helped us understand how to fix the Unexpected HTTP 403 error in Java Examples error.
How do I fix error 403 in Java?
Double Check the Address. The most common reason for a 403 error is a mistyped URL. First, ensure that the address you are trying to access is for a web page or file, not a directory. For example, a regular URL would end in .com, .
How do I fix HTTP 403 forbidden?
How to Fix the 403 Forbidden Error
- Check the . htaccess File.
- Reset File and Directory Permissions.
- Disable WordPress Plugins.
- Upload an Index Page.
- Edit File Ownership.
- Verify the A Record.
- Scan for Malware.
- Clear Your Web History/Cache.
What causes HTTP 403 Forbidden error?
The 403 Forbidden error appears when your server denies you permission to access a page on your site. This is mainly caused by a faulty security plugin, a corrupt . htaccess file, or incorrect file permissions on your server.20-Jul-2021
What HTTP 403 means?
An HTTP 403 response code means that a client is forbidden from accessing a valid URL. The server understands the request, but it can’t fulfill the request because of client-side issues. API Gateway APIs can return 403 responses for any of the following reasons: Issue. Response header.
Why am I getting 403 error the request could not be satisfied?
The error message «403 Error — The request could not be satisfied. Request Blocked.» is an error from the client. This error can occur due to the default actions of AWS WAF rules associated with the CloudFront distribution.13-Jun-2022
Can you bypass a 403 error?
How to Bypass 403 restrictions? There are many headers and paths which you can use to bypass 403 restrictions. Adding Headers in request :By adding different headers in request with value 127.0. 0.1 can also help in bypassing restrictions.03-Oct-2021
Is Error 403 a ban?
As mentioned in the intro, HTTP 403 forbidden error means the server acknowledges the request but refuses to process it. This status is much like the 401 code. The difference is that re-authentication will not produce positive results with the 403 code. Access is completely banned.20-May-2022
Will Error 403 go away?
A 403 Forbidden Error occurs when you do not have permission to access a web page or something else on a web server. It’s usually a problem with the website itself. However, you can try refreshing the page, clearing your cache and cookies, and disconnecting from any VPN you might be using.18-Oct-2022
How do I fix a 403 error in Chrome?
You can go with some steps which may fix this issue:
- Refresh the Webpage.
- Crosscheck the URL.
- Clear your Browers Cookies and Cache.
- Disable Unknown Chrome Extensions.
- Try to open the page in Incognito Mode.
- You may find a login panel where you need to enter the login details to access the page.
- Contact the Website Owner.