Get a search engine website for yourself.

Written by Mo

Find the project here GitHub-Search Engine https://github.com/mohamedsbadawy/scholar-engine

Research Engine

Research Engine is a Python-based web application designed to help users easily find and organize academic articles, journals, and related research. It leverages the Scrapy framework for web scraping and Flask for the web interface. This tool allows users to search for scholarly articles, view related references and citations, and save results into a personal library. Users can also download their saved search results and explore connected maps of their research findings.

live view.

You can also access the live application at Related Engine.

Features

Scholarly Search: Search for articles, journals, and papers related to specific keywords.
Related References and Citations: Automatically fetch related references and citations for the searched articles.
User Accounts: Users can create accounts to save their search results, manage their personal library, and download saved data.
Connected Maps: Visualize the relationships between articles and references through interactive connected maps.
Review Section: A modifiable section where users can input reviews for articles, journals, or any other related content.
SQL database integrations: to use user registration, and Journals listings, you need to connect your SQL database according to the instructions below.

Installation

To install and run the Research Engine on your local machine, follow these steps:

Clone the Repository:

 git clone https://github.com/mohamedsbadawy/scholar-engine.git
 cd research-engine

Create a Virtual Environment (optional but recommended):

 python -m venv venv
 source venv/bin/activate  # On Windows: venv\Scripts\activate

Install the Required Packages: Install all dependencies using the requirements.txt file:
```
 pip install -r requirements.txt
```
Change SQL DB name and username: Change SQL DB name and username: in /website/init.py you need to input your DB username and passwords to use the SQL functinality. You can disable these features by commenting out the lines that controls DB, and User functionality.
```
 app.config['SQLALCHEMY_DATABASE_URI'] = f'mysql+pymysql://SQLName:{encoded_password}@localhost:3306/SQLUserName'
 app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False  # Disable Flask-SQLAlchemy modification tracking
    
```

Rudimentary admin page to add Journal enteries: admin page to add Journal enteries: in /website/scijournals.py change userid here to whatever userid you choose to be the admin

 # admin
 @jour.route('/admin/', methods=['GET', 'POST'])
 @login_required
 def import_data():
     # Get the current user
     if current_user is None:
         flash('please log in')
         return redirect(url_for('views.home'))
         # Check if the user is authenticated as user ID 2
     elif current_user.id != 2:
         flash('You don not have access',category='error')
         return redirect(url_for('views.home'))
     return render_template('admin.html')

To allow the app to send notification emails: To allow the app to send notification emails: in /website/init.py

 # Configure Flask-Mail
 app.config['MAIL_SERVER'] = 'relengine.com'  # Replace with your SMTP server address
 app.config['MAIL_PORT'] = 465  # Replace with your SMTP server port
 app.config['MAIL_USE_TLS'] = False
 app.config['MAIL_USE_SSL'] = True
 app.config['MAIL_USERNAME'] = 'email username'  # Replace with your email address
 app.config['MAIL_PASSWORD'] = 'Password'  # Replace with your email password
 mail.init_app(app)

Sematic Scholars API: To use "AI-Search Engine": obtain an API from sematicscholars, and add it to API.Py.
Run the Web Application: Start the Flask application by running:
```
 flask run
```
By default, the application will be accessible at http://127.0.0.1:5000.
Access the Application:

Open your web browser and navigate to http://127.0.0.1:5000 to start using the Research Engine.

Usage

Search: Enter a keyword or phrase into the search bar to find relevant scholarly articles.
View Results: Browse through the search results, which include related references and citations.
Save to Library: Logged-in users can save articles to their personal library for future reference.
Download Results: Users can download their saved search results in a convenient format.
Connected Maps: Explore the relationships between articles and references using the interactive map feature.
Review Journals: Contribute by reviewing articles or journals, enhancing the collaborative academic environment.
Job hub: Contribute by reviewing articles or journals, enhancing the collaborative academic environment.
SQL integration: Contribute by reviewing articles or journals, enhancing the collaborative academic environment.

Reused codes

Some of the code in the project was open source and was repurposed to the current use case.

Contributing

Contributions are welcome! If you would like to contribute to this project, please fork the repository and submit a pull request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Related Engine - Blog