(L-R) James B. Jones, Robert F. Kennedy Jr. Photo:

Robert B. Jones/instagram; Mario Tama/Getty

• James B. Jones responds to Robert F. Kennedy Jr.’s comments about autism in an exclusive interview with PEOPLE
• The Love on the Spectrum star slams Kennedy’s generalizations about autism
• He shares what he ultimately would want people to know about autism

For James B. Jones, one of the stars of Netflix’s hit show Love on the Spectrum, Robert F. Kennedy Jr.‘s comments about autism are nothing to take lightly.

James, who made things official with his girlfriend, Shelley Wolfe, on the show’s season 3 finale, tells PEOPLE that after the secretary of health and human services made derogatory remarks about autism during his first press conference with his new title, he was “not particularly pleased.”

“They clearly were made from a position of extreme ignorance. Autism manifests itself differently in every person. No two people who have it are exactly the same,” James says, calling Kennedy’s remarks “flagrantly false.”

In his controversial remarks, which were made in response to a new study by the CDC that found 3% of children in the U.S. have autism, Kennedy made sweeping claims about the livelihood of people with autism.

Robert F. Kennedy Jr. Michael M. Santiago/Getty

“These are kids who will never pay taxes. They’ll never hold a job. They’ll never play baseball. They’ll never write a poem. They’ll never go out on a date. Many of them will never use a toilet unassisted,” he said.

In response, James tells PEOPLE that “one should not apply a blanket approach or make generalized presumptions about people who have autism.”

While he admits that it took “slightly longer” to develop his social skills, and it was something he had to “work harder” and “put forth greater effort in developing and honing,” he has succeeded in doing so.

“I have a permanent 40-hour per week job with benefits, which I have held steadily since 2017 and am in no danger of losing. I can operate a vehicle with no issues. I pay all of my own expenses,” he shares.

James and Shelley on Love on the Spectrum. Netflix

“Other than the fact that I live with my parents still, I believe I’m doing quite a good job being self-sufficient. And the fact that I do not yet have my own house at age 37 is due to factors that are beyond my control,” says James, who notes that he also graduated from college with a 3.85 GPA.

“I did very well in that. So yes, I have not struggled academically at all. I pay my bills. I pay my taxes. I drive a car,” he says.

“Look where I am now. I am a star of a major Netflix show. I have a huge relationship with a woman whom I love very much, who loves me very much, and everything’s going great for me in my life right now,” continues James.

Further denouncing Kennedy’s comments, James says that “it is not that there are more people with autism, it is that doctors have become more skilled in identifying traits of autism, and that society has become more accepting of autism overall.”

James and Shelley appear in Love on the Spectrum. Netflix

“I would hate to see all that progress be lost,” he says.

However, he recognizes that “unfortunately, there are still some people who do not have a thorough understanding of those who are different from themselves.”

He considers himself “very fortunate” that “the majority of people whom I have met understand autism or at least make an effort to understand it.”

When asked what he ultimately wants people to know about autism, James says, “We need to be accepting of each other’s differences and understand that no two people ever have the same experience.”

Shelley and James appear in Love on the Spectrum. Netflix

“The fact that someone’s different from oneself is not necessarily a bad thing and does not necessarily make them a bad person,” he adds. “So we need to be patient and understanding of each other.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

Love on the Spectrum season 3 is streaming on Netflix.